Deciphering Stereotypes in Pre-Trained Language Models (2023)

Abstract: Warning: This paper contains content that is stereotypical and may be upsetting. This paper addresses the issue of demographic stereotypes present in Transformer-based pre-trained language models (PLMs) and aims to deepen our understanding of how these biases are encoded in these models. To accomplish this, we introduce an easy-to-use framework for examining the stereotype-encoding behavior of PLMs through a combination of model probing and textual analyses. Our findings reveal that a small subset of attention heads within PLMs are primarily responsible for encoding stereotypes and that stereotypes toward specific minority groups can be identified using attention maps on these attention heads. Leveraging these insights, we propose an attention-head pruning method as a viable approach for debiasing PLMs, without compromising their language modeling capabilities or adversely affecting their performance on downstream tasks.

## introduction stereotypes, serving as simplified and generalized representations of societal beliefs, have turned into a challenging issue in the sphere of natural language processing (nlp). their inadvertent encoding in pre-trained language models (plms) and propagation in downstream applications has incited concerns about the fairness and bias of such systems @xcite @xcite . to build unbiased language technologies, it is crucial to understand how these models encode and detect stereotypes thoroughly. a significant body of research has demonstrated the presence of such biases and worked on methods to effectively detect them in plms. however, these methods fail to provide an understanding of the processes underlying stereotype encoding and detection in plms. current approaches for examining stereotypes in plms demand intricate human knowledge about these stereotypes and entail careful manual curation of examples @xcite . these characteristics make such approaches costly and time-consuming to implement. furthermore, they lack the capability to detect newlyemerged or complicated stereotypes, such as intersectional stereotypes, thereby highlighting the need for a more comprehensive, adaptable, and less labor-intensive approach. with a focus on addressing these limitations, we examine the intricate relationship between stereotype encoding and detection within plms. we propose a framework for examining stereotypes in transformer-based plms by conducting attentionhead probing. the motivation for using probing arises from the inherent complexity of transformer models, which hampers their theoretical analysis, and the widespread acceptance of probing methods as an effective tool to approximate the functioning of these models in natural language understanding @xcite . we further use shapley values @xcite to quantify the individual contributions of different attention heads in stereotype detection, thereby shedding light on their roles in encoding stereotypes. this integration helps in the systematic handling of potential interactions among attention heads, thereby allowing us to quantify the individual contribution of each head effectively. our approach aims to reduce the reliance on manually-curated word-level stereotypical instances by combining stereotype detection research with the examination of stereotypes in plms. this merger paves the way for using sentence-level stereotype detection datasets that are more easily annotated. further, our framework facilitates a detailed analysis of stereotypes toward specific minority groups within each plm by integrating attention analysis with shap, a perturbation-based model interpretation method. through a series of carefully designed exper-iments, we uncovered that a substantial portion, ranging from 15% to 30%, of attention heads in six transformer-based models of various sizes and architectures significantly impact the models' ability to detect stereotypes. this considerable number of influential attention heads underlines the depth and complexity of stereotypes representation in these models. the models we examined cover a broad range of architectures, including encoderonly models such as bert @xcite and roberta @xcite , as well as encoderdecoder models such as t5 @xcite and @xcite . our experiments revealed a significant correlation between the attention heads that play a crucial role in detecting stereotypes and those involved in encoding these stereotypes. notably, our attentionhead ablation experiments demonstrated that these heads exhibited a distinct preference for stereotypical expressions when processing language. such a finding not only underscores the robustness of our approach, which uses stereotype detection datasets to explore intrinsic biases in plms but also offers a practical and efficient strategy for debiasing these models through targeted attention-head pruning. our framework also offers a more granular perspective by breaking down stereotypes by types and targets. we analyzed the behavior of the most impactful attention heads, as identified by our probing experiments, using shap and attention analyses. this allowed us to examine stereotypes exhibited by each plm toward five frequently stereotyped groups: aged people, females, muslims, african people, and middle-eastern people. our detailed analysis uncovered a shared set of common stereotypes across all the plms. however, we also observed distinct stereotypical expressions associated with each minority group within different plms. this discovery underscores the necessity of leveraging diverse instances when evaluating stereotypes in different plms for fair assessments and effective stereotype reduction -an aspect that current research has often overlooked. the robustness of our results, which hold across variations in random-seed selection, dataset utilization, and plm and checkpoint choices, further validates our approach and attests to the reliability of our findings. perhaps most importantly, our analyses can be effortlessly extended to other plms or stereotypes with other types or targets, eliminating the need for pre-assumed knowledge about how stereotypes are expressed in the text. this attribute greatly enhances the versatility and practicality of our approach in the quest for stereotype-free plms. ## background and our contributions historically, the exploration of intrinsic biases within plms has predominantly revolved around comparison-based methods. these methods involve contrasting the propensity of plms to generate stereotypical versus non-stereotypical content from similar prompts @xcite . for instance, @xcite probe into bert's gender bias by comparing its likelihood of associating a list of professions with pronouns of either gender. similarly, @xcite delve into social bias by contrasting the probabilities of bert and roberta, associating adjectives that describe different stereotype dimensions with various groups. it would be even more costly to design multiple stereotypical contents for each minority group and annotate sentence pairs accordingly for identifying and assessing stereotypes in different plms. despite their utility, such assessment methodologies come with inherent limitations, particularly when applied to the task of debiasing plms. firstly, they necessitate costly parallel annotations (comprising both stereotypical and non-stereotypical sentences pertaining to an identical subject), which are not readily expandable to accommodate emerging stereotypes. as @xcite noted, stereotypes evolve in tandem with cultural shifts, making it critical to align stereotype analysis with current definitions and instances. secondly, the assumption that stereotypes are solely attributable to specific word usage oversimplifies the complex nature of biases. plms might favor particular words not because of inherent biases but due to their contextual prevalence. this complexity, coupled with the implicit nature of biases @xcite , challenges the efficacy of existing stereotype assessment approaches. lastly, prevailing stereotype-evaluation benchmarks assume uniform types of stereotypes across all plms, an assumption that is not necessarily valid. designing multiple stereotype instances for each minority group and annotating corresponding sentence pairs would impose an even greater cost. in this paper, we propose an innovative approach that bridges the gap between the assessment of stereotypes encoded in plms and the models' stereotype detection capabilities. our approach leverages datasets that are more readily annotated and modified, as they do not demand parallel annotations or preconceived stereotypical expressions. furthermore, our method operates at the sentence level rather than the word or phrase level, facilitating more versatile evaluations of stereotypical expressions or implicit stereotypes. to address the existing research and dataset gap on implicit stereotypes, we introduce a manually-validated implicit stereotype dataset generated using chatgpt. our framework enables the identification of stereotypical expressions within each plm and the examination of the similarities and differences in how stereotypes are encoded across various plms, enabling a deeper understanding of the unique stereotypical tendencies inherent in different models. ## datasets for stereotype examination our study deploys three extensively used datasets to investigate stereotypes in plms: (1) stereoset @xcite , (2) crows-pairs @xcite , and (3) winobias @xcite . however, these existing datasets are not without issues, including the occasional unnaturalness of sentences constructed via word substitution in sentence pairs and the presence of instances that are incorrectly classified or not truly stereotypical @xcite . moreover, these datasets tend to oversimplify stereotypes by restricting examination to short sentences where stereotypes are expressed explicitly through a few words syntactically tied to the subject (i.e., explicit stereotypes). our investigations reveal that existing datasets on stereotypes often conflate stereotype representation with negative sentiments or emotions. this conflation limits nuanced analysis of stereotypes, as it reduces their complexity to mere emotional charge. for a comprehensive discussion of these limitations, please refer to appendix a.1. to address this shortcoming, we introduce a novel dataset focused on "implicit stereotypes" foot_0 . this dataset is generated using chatgpt, which facilitates the extraction of more subtle and contextually embedded stereotypes resembling those found in natural language dialogues. we use large language models like chatgpt for dataset construction primarily because they possess extensive training data, encompassing real-world conversations and online text. this enables them to generate implicit stereotypes with relative ease compared to human annotators, who may find the task more challenging. while the dataset constructed for this paper is limited in its scope, the methodology can be scaled to create more expansive datasets covering a broader range of minority groups and stereotypes. the dataset generation procedure comprises three main steps: (1) initially, chatgpt is queried to generate a list of common stereotypes associated with 17 demographic groups, using the prompt specified in appendix figure a2a . (2) in the subsequent stage, each target demographic group and its corresponding stereotypes are fed into chatgpt. the model is instructed to produce five instances of implicit stereotypes for each group-stereotype pairing, utilizing the prompt detailed in appendix figure a2b . (3) finally, chatgpt is employed to de-bias each generated example, resulting in a set of 425 non-stereotypical instances. these instances undergo manual verification to ensure their quality and fidelity. following the removal of duplicate and noisy entries through manual curation, we assembled the implicitstereo dataset. this dataset comprises 416 stereotypical and 374 non-stereotypical instances, targeting 17 distinct demographic groups. we conducted manual validation to assess the dataset's quality by randomly sampling 100 pairs of stereotypical and de-biased instances. validators were tasked with determining whether each stereotypical instance overtly presented prevalent stereotypes about a given demographic group and whether its de-biased counterpart was entirely free of such stereotypes. in 95 out of 100 sampled instances, at least two out of three validators concurred that the instances accurately encapsulated common stereotypes toward the demographic groups in question. for 86 out of the 100 instances, a similar majority agreed that all stereotypes were effectively removed in the de-biased samples without affecting the unrelated content. inter-annotator agreement for these validation tasks was quantified using fleiss' κ @xcite , with scores exceeding 0.74 for both sets of validations. this attests to the high quality of the implicitstereo dataset. given these promising results, the implicitstereo dataset has been incorporated into our primary experiments alongside three publicly available datasets. examples from all datasets are provided in dataset instance label the chess player was asian. stereotypical stereoset the chess player was hispanic. anti-stereotypical asians cant drive because they have slanted eyes. / white people cant drive because they have slanted eyes. stereotypical crows-pairs whites are not good at reading. / hispanics are good at reading. anti-stereotypical the [developer] argued with the designer because [he] did not like the design. stereotypical winobias the [developer] argued with the designer because [she] did not like the design. anti-stereotypical [target: muslim] you're muslim and you want to be a chemist? that's an interesting choice. stereotypical implicitstereo [target: muslim] you're a muslim and interested in becoming a chemist? that's a fascinating career path. non-stereotypical in the training, evaluation, and probing of stereotype detection models, we utilize sentencelevel stereotypical vs. anti-stereotypical (or nonstereotypical) annotations instead of sentence pairs. these labels are readily available as ground-truth annotations in stereoset, winobias, and our im-plicitstereo dataset, and we employ them in their original form for the stereotype detection task. crows-pairs provides a pair of sentences, one more stereotypical and the other less so, for each stereotypical or anti-stereotypical instance. for the stereotype detection task, we label the more stereotypical sentence of stereotypical instances as stereotypical and the less stereotypical sentence of anti-stereotypical instances as anti-stereotypical. stereoset features annotations for four types of stereotypes: gender, race, religion, and profession. crows-pairs covers a broader spectrum, encompassing race/color, gender/gender identity or expression, socioeconomic status/occupation, nationality, religion, age, sexual orientation, physical appearance, and disability. winobias focuses on gender biases that influence pronoun choice (e.g., he vs. she) about 40 types of professions. implicit-stereo contains stereotypes under five categories: age (young or old), gender (male, female, or genderqueer), ethnicity (black, white, asian, hispanic), country of origin (french, american, arabic, in-dian, middle eastern), and religion (christian, muslim, jewish). we conduct separate analyses of the stereotypes in plms toward specific minority groups within each category in our textual-clue examinations. we split stereoset, crows-pairs, and implicit-stereo into training (80%) and testing (20%) sets. the training portions of stereoset, crows-pairs, and implicitstereo, along with the development set of winobias, are employed for probing and fine-tuning stereotype detection models. the test portions of all datasets are reserved for evaluating the intrinsic biases present in plms. the metric in the probing and ablation experiments is accuracy. ## models for stereotype examination our research is focused on investigating the encoding and detection of stereotypes in transformerbased plms. to investigate the influence of different pre-training corpora and objectives on the biases present in plms, our main experiments utilize the bert and roberta models. despite being similar in size, these two models have been pre-trained using distinct corpora and objectives, offering valuable contrast in our study foot_1 . to ensure the robustness and general applicability of our findings, we further examine four other models. these include both the small and base versions of t5 and flan-t5, thereby covering a broader range of model architectures. we use the huggingface implementation of all the models @xcite . ## understanding attention heads' role in stereotype detection this section dives into the role of attention heads in transformer-based plms for detecting stereo-types in the text. we utilize a shapley-value-based probing approach to discern their contributions. ## stereotypical encoding by plms here, we analyze the relationship between the attention-head contributions in both the detection and encoding of stereotypes within plms. we use the stereotype score (ss) @xcite , to gauge the level of stereotyping within plms, and the language modeling score (lms) to measure their linguistic proficiency. building on nadeem et al. ## analyzing textual clues here, we undertake an analysis of the textual clues that significantly influence the inferences of inherent stereotypes within each plm. this examination arises from our previous experimental findings that the same attention heads are crucial to both encoding and detecting stereotypes. we focus on analyzing the most attended single words and word pairs in the top-5 attention heads as revealed by our probing experiments, averaging the attention across these heads. we use shap to reduce potential noise (such as high attention scores on function words or special tokens). we use the attention-head contributions obtained from implicitstereo to avoid biasing the textual analyses since implicitsteroe does not pre-assume any word to be always stereotypical toward a group. we combine all four datasets, selecting instances related to five minority groups (specifically, the elderly, females, muslims, african, and middle-eastern people) for our textual-clue analysis. for illustrative purposes, we present the singleword textual analysis results on bert and roberta in figure 3 . we reserve the complete set of results for appendix e. our analyses reveal that the top-10 word-level textual clues align with common stereotypes towards the five minority groups. for instance, words like "fat", "bald", and "bankrupt" stand out for age stereotypes; "gossip" and "silly" for gender stereotypes; "violent", "kill", and "terrorists" for religious (muslims) and racial (middle-eastern people) stereotypes; and "dirty", "crime", and "crude" for racial (african people) stereotypes. this is consistent with prior studies on the prevalence of stereotypes about minority groups in plms. for example, @xcite discovered that gpt-3 @xcite generates stereotypical associations of muslims with phrases like "shooting at will" and "bombing." our word-pair analyses also present similar rankings but with word pairs usually expressing stronger and more stereotype-laden meanings, such as "all terrorists" for muslims, implying all muslims are terrorists. the anti-stereotypical words and word pairs we find are mostly antonyms of the associated stereotypical words, such as "nice" and "caring," which these models use to identify anti-stereotypes towards middle-eastern people. we also note some variability across the models in our study, despite many models highlighting common stereotypical words and word pairs. one potential reason for these differences could be the size of the models, as larger models might capture more instances of co-occurrence between stereotypical words and minority group mentions. for instance, t5-base and flan-t5-base rank "alcohol" and "urine" highly for the elderly, whereas their smaller counterparts do not. variations can also stem from differences in pre-training corpora and objectives between plms with the same architecture and structure. for example, bert highlights "violent" and "pirates" for african people and "bombs" for middle-eastern people, while roberta does not. similarly, roberta ranks "bankrupt" and "gross" much higher than bert for stereotypes towards the elderly. these findings indicate that different plms might harbor diverse stereotypes towards minority groups. therefore, using the same dataset to assess stereotype levels across all plms might underestimate certain stereotypes. the combined application of our probing technique and textual-clue analysis framework could aid in identifying the most pronounced stereotypes within each plm. ## conclusion and future work in this paper, we sought to deepen the understanding of the connection between the encoding and detection of stereotypes within plms. we performed extensive probing and ablation studies and, informed by the results, developed a framework to explore the intrinsic stereotypes within each plm. this framework leverages both textual and attention analyses and solely relies on stereotype detection annotations. our study unveils that stereotypes are not uniformly distributed across different plms. this highlights the need for model-specific stereotype assessment datasets and tailored debiasing techniques. our framework introduces an efficient means of debiasing plms without restructuring or retraining and without the need for expensive pairwise stereotype/anti-stereotype annotations. future research can potentially merge our methods with other plm debiasing techniques. 11336 this study has certain limitations. firstly, our experiments were primarily conducted in english due to the scarcity of stereotype assessment datasets in other languages. while our framework is capable of handling complex scenarios such as intersectional stereotypes, we were unable to explore these due to a lack of adequately annotated datasets. we also constructed an implicit stereotype dataset using chatgpt alongside three publicly available datasets. we pursued this approach because existing stereotype examination datasets often oversimplify the task and have known quality concerns, as indicated by @xcite . our dataset addresses several issues of prior ones, such as unnatural phrasing, overly explicit stereotype expression, and excessive intertwining of stereotypes with negative emotions and sentiments. nonetheless, we recognize potential noise in the chatgpt-generated data, despite our careful manual curation and high-quality human validation results via amazon mechanical turk. unvalidated data points may still contain biases or incorrect information from chatgpt, which could influence our results. further, our dataset doesn't completely rectify the problems with existing datasets. however, we view implicitstereo as an additional data source, providing a more equitable examination of stereotypes in plms, and any potential biases from implicitstereo should not significantly impact our analyses.