Bias A-head? Analyzing Bias in Transformer-Based Language Model Attention Heads (2025)

Abstract: Transformer-based pretrained large language models (PLM) such as BERT and GPT have achieved remarkable success in NLP tasks. However, PLMs are prone to encoding stereotypical biases. Although a burgeoning literature has emerged on stereotypical bias mitigation in PLMs, such as work on debiasing gender and racial stereotyping, how such biases manifest and behave internally within PLMs remains largely unknown. Understanding the internal stereotyping mechanisms may allow better assessment of model fairness and guide the development of effective mitigation strategies. In this work, we focus on attention heads, a major component of the Transformer architecture, and propose a bias analysis framework to explore and identify a small set of biased heads that are found to contribute to a PLM’s stereotypical bias. We conduct extensive experiments to validate the existence of these biased heads and to better understand how they behave. We investigate gender and racial bias in the English language in two types of Transformer-based PLMs: the encoder-based BERT model and the decoder-based autoregressive GPT model, LLaMA-2 (7B), and LLaMA-2-Chat (7B). Overall, the results shed light on understanding the bias behavior in pretrained language models.

## introduction transformer-based pretrained language models such as bert @xcite , @xcite , and large foundation models such gpt-3 @xcite , palm @xcite , and llama @xcite have achieved superior performance in many natural language processing (nlp) tasks @xcite @xcite @xcite . however, since plms and foundation models are trained on large human-written corpora, they often encode undesired stereotypes towards different social groups, such as gender, race, or people with disabilities @xcite @xcite . for example, gpt-2 has been shown to generate stereotypical text when prompted with context containing certain races @xcite . a stereotype is an over-simplified belief about a particular group of people, e.g., "women are emotional." stereotyping can cause representational harms @xcite because it can lead to discrimination, prejudice, and unfair treatment of individuals based on their membership in a particular group @xcite . in order to design robust and accountable nlp systems, a rich and growing body of literature has investigated the stereotypes in plms from two perspectives. the first line of work aims to quantify the stereotypical biases. for example, @xcite propose a sentence encoder association test (seat), and @xcite develop the stereoset dataset to assess if a plm encodes stereotypes. the second line of work aims to propose de-biasing strategies that remove undesired stereotypical association biases from plms @xcite @xcite . similarly, foundation models also need to be further aligned to alleviate its bias concern, using techniques such as reinforcement learning from human feedback (rlhf) @xcite . we later demonstrate that rlhf can help reduce biases by comparing llama-2 with llama-2-chat. however, there are still gaps in understanding stereotypical biases in transformerbased language models. for bias assessment, while the common practice uses one score to quantify the model bias, it is unclear how the bias manifests internally in a language model. for bias mitigation, existing works are usually designed in an end-toend fashion with a "bias neutralization" objective, but the inner-workings of the entire debiasing procedure remain a black-box. there is a need for in-depth analysis that uncovers how biases are encoded inside language models. we propose a framework to analyze stereotypical bias in a principled manner. 1 our main research question is, how does bias manifest and behave internally in a language model? prior work in better understanding the internal mechanisms of deep neural networks has focused on specific model components. for example, we take inspiration from the seminal work of finding a single lstm unit which performs sentiment analysis @xcite and attributing types of transformer attention heads as "induction heads" that do in-context learning @xcite . in this work, we focus on attention heads in pretrained language models. attention heads are important because they enable transformer-based models to capture relationships between words, such as syntactic, semantic, and contextual relationships @xcite . our proposed framework begins by measuring the bias score of each transformer attention head with respect to a type of stereotype. this is done by deriving a scalar for each attention head, obtained by applying a gradient-based head importance detection method on a bias evaluation metric, i.e., the sentence encoder association test (seat, @xcite . heads associated with higher bias scores are dubbed biased heads, and are the heads upon which we then conduct in-depth analyses. in our analysis, we start by investigating how gender biases are encoded in the attention heads of bert. we visualize the positions of biased heads and how they are distributed across different layers. to further verify that the identified biased heads indeed encode stereotypes, we conduct a counterstereotype analysis by comparing the attention score changes between the biased heads and normal (non-biased) heads. specifically, given a sentence containing a gender stereotype such as "women are emotional," we obtain its counter-stereotype "men are emotional." we then calculate the attention score change for the stereotypical word "emotional." since the only difference between the original sentence and its counter-stereotype sentence is the gender-related word, we would expect significant score changes for those heads that encode biases, and minimal changes for those heads that do not encode biases. our analysis on a large external corpus verifies that the attention score change of the biased heads are statistically and significantly greater than that of the normal heads. later in the paper, we extend the analysis to investigate bias in the gpt model, llama-2, llama-2-chat, as well as racial stereotype associated with caucasians and african americans. moreover, we show that a simple debiasing strategy that specifically targets a small set of biased heads (by masking), which is different from previous end-to-end bias mitigation approaches that tune the entire plm, yields a lower model bias performance with minimal disruption to language modeling performance. in summary, this work makes two important contributions. first, we open the black-box of plm biases, and identify biased heads using a gradient-based bias estimation method and visualizations, shedding light on the internal behaviors of bias in large plms. the proposed framework also contributes to the literature on understanding how plms work in general @xcite . second, we propose a novel counter-stereotype analysis to systematically study the stereotyping behavior of attention heads. as a resource to the research community and to spur future work, we open-source the code used in this study at https://github.com/hduanac/biased-head/ . ## attention head bias estimation framework our proposed framework for attention head bias estimation measures the bias score of transformer self-attention heads with respect to a focal/concerning bias (e.g., gender). we first introduce a new variable, the head mask variable (boolean), that exists independently in each attention head. we then discuss how this variable can be utilized to quantify the bias in each attention head. @xcite propose a network pruning method that examines the importance of each selfattention head in a transformer model. given our interest in measuring the importance of each self-attention head with respect to a concerning bias, for each attention layer i comprised of h attention heads, we introduce a variable m @xmath0,1 , m i,2 , . . . , m i,h ] ′ called the head mask variable that is multiplied element-wise with the output from each attention head in the ith layer. this allows us to understand (and control) the contribution of each attention head to the model's final output: ## experimental setup gender and racial bias word lists: our analysis focuses on studying gender bias and racial bias, which are two of the most commonly examined stereotypes in plms. for gender bias, we employ attribute and target word lists used in prior literature @xcite . in total, the gender attribute word list contains 444 unique words (222 pairs of feminine-masculine words), and the target list contains 84 gender related stereotypical words. 4 for racial bias, we examine the stereotypical association between caucasian/african american terms and stereotypical words. specifically, we use the attribute word list and target word list proposed in prior work @xcite . the racial attribute word list contains 6 unique words (3 pairs of african-american vs. caucasian words), and the target list contains 10 racial stereotypical words. 5 external corpus for bias estimation: we use the news-commentary-v15 corpus to obtain contextualized word embeddings for plms and identify biased heads using the bias estimation method (sec. 3.2). this corpus has often been used in prior plm bias assessment and debiasing work @xcite . 6 plms: we study the encoder-based bert model, the decoder-based gpt model, llama-2, and llama-2-chat. for the bert model, we consider bert-base, which is comprised of 12 transformer layers with 12 heads in each layer. for the gpt model, we consider gpt-2 small @xcite , which also consists of 12 transformer layers with 12 attention heads in each layer. we consider llama-2 (7b) @xcite and its finetuned version llama-2-chat, which consists of 32 transformer layers with 32 attention heads in each layer. 7 we implemented the framework and conducted experiments on an nvidia rtx 3090 gpu using pytorch 1.9. plms were implemented using the transformers library. 8 5 assessing gender bias in bert and gpt prior literature has shown that plms like bert and gpt exhibit human-like biases by expressing a strong preference for male pronouns in positive contexts related to careers, skills, and salaries @xcite . this stereotypical association may further enforce and amplify sexist viewpoints when the model is fine-tuned and deployed in real-world applications such as hiring. we use the proposed method to assess gender bias in bert and gpt-2. ## assessing racial stereotyping in this section, to demonstrate our bias analysis framework is also applicable to other types of biases beyond gender bias, we apply our framework to examine racial bias between caucasian/african american terms and racial related stereotypical words such as criminal, runner, etc. in the following experiment, we use bert-base as the underlying plm. 13 we visualize the bias score distribution and heat map in figure 1c and figure 2c respectively. much like the distribution of gender bias in bert, we observe several heads with significantly higher bias scores. moreover, the biased heads appear across all layers; some of the highest scores are distributed in the higher layers. we conduct a counter-stereotype experiment to validate the identified racial biased heads. similar to the counter-stereotype experiment step for gender bias analysis, we first obtain a set of sentences from the reddit corpus that contains both the racial attribute words (such as "black") and stereotypical words (such as "criminal"). then we measure the attention score change in a sentence and its counterfactual by replacing an attribute word to its counterpart word (such as "white"). figure 4c shows that for the bias heads, the average attention score is significantly lower in the counter-stereotype group than in the original group, indicating these heads encode stronger racial stereotype associations (t-@xmath1.324, p-@xmath2.05, @xmath3. in contrast, for the unbiased heads group, there is no statistical difference in the original sentences and their counter-stereotypes (t-@xmath4.107, p-@xmath5.54, @xmath6. ## generalizing to large language models (llms) we generalize our bias analysis framework to llms -specifically, llama-2 (7b) and its instruction-tuned counterpart llama-2-chat (7b) @xcite . we repeat the same procedures, as done in the earlier experiments, to assess gender bias. the obtained bias scores for llama-2 and llama-2-chat are 0.27 and 0.18, respectively, suggesting that instruction-tuned llms exhibit less biases as compared to its base model. this is potentially due to the rlhf process that mitigates the stereotypes in llms through human feedbacks. the respective bias score distribution appears in appendix d, as expected, we observe llama-2-chat contains significantly less heads with pronounced positive bias scores relative to the base version. ## conclusion and discussion in this work, we present an approach to understand how stereotyping biases are encoded in the attention heads of pretrained language models. we infer that the biases are mostly encoded in a small set of biased heads. we further analyze the behavior of these biased heads, by comparing them with other regular heads, and confirm our findings. we also present experiments to quantify gender bias and racial bias in bert and gpt. this work is among the first work aiming to understand how bias manifests internally in plms. previous work has often used downstream tasks or prompting to examine a plm's fairness in a black-box manner. we try to open up the black-box and analyze different patterns of bias. in doing so, we strengthen our understanding of plm bias mechanisms. future work can apply our method to assess concerning biases in increasingly large foundation models. overall, our work sheds light on how bias manifests internally in language models, and constitutes an important step towards designing more transparent, accountable, and fair nlp systems.