Identifying and Adapting Transformer-Components Responsible for Gender Bias in an E nglish Language Model (2023)

Abstract: Language models (LMs) exhibit and amplify many types of undesirable biases learned from the training data, including gender bias. However, we lack tools for effectively and efficiently changing this behavior without hurting general language modeling performance. In this paper, we study three methods for identifying causal relations between LM components and particular output: causal mediation analysis, automated circuit discovery and our novel, efficient method called DiffMask+ based on differential masking. We apply the methods to GPT-2 small and the problem of gender bias, and use the discovered sets of components to perform parameter-efficient fine-tuning for bias mitigation. Our results show significant overlap in the identified components (despite huge differences in the computational requirements of the methods) as well as success in mitigating gender bias, with less damage to general language modeling compared to full model fine-tuning. However, our work also underscores the difficulty of defining and measuring bias, and the sensitivity of causal discovery procedures to dataset choice. We hope our work can contribute to more attention for dataset development, and lead to more effective mitigation strategies for other types of bias.

## introduction modern neural language models exhibit social biases, such as biases based on gender, religion, ethnicity and other protected attributes. these biases may lead to real harms when used in downstream applications (e.g. @xcite . detecting and mitigating biases in language models has therefore become an important area of research. early detection methods relied on lists of words to measure associations with e.g., specific genders (e.g. @xcite . most current detection * shared senior authorship. methods work with curated sets of sentence pairs or triplets, and measure differences in sentence probabilities or anaphora resolution probabilities (e.g. @xcite @xcite . proposed mitigation strategies include targeted changes to the training data (e.g., cda; @xcite , training procedure (e.g., adversarial learning; @xcite , model parameters (e.g., inlp; @xcite , or language generation procedure (e.g., "self-debiasing"; @xcite . despite this work, we still lack a proper understanding of how to best measure biases (how do we guarantee the representativeness for real-world harm of a set of sentence pairs, or of a linguistic phenomenon such as anaphora resolution?), how biases are implemented in the language model internals (is there a unified locus, or is, e.g., gender bias the aggregate effect of many independent model decisions?), and what techniques are effective at reducing undesirable downstream behavior (e.g., is data curation more or less effective than filtering output? is intervening in the model internals feasible?). empirically, success in detecting and mitigating biases depends on many factors, including the choice of embeddings, training regimes, data sets and model choices @xcite @xcite @xcite . the "black-box" nature of lms makes it difficult to identify and interpret how bias manifests and propagates in them, especially relying solely on correlational methods. the starting point for the current paper is the intuition that if, instead, it were possible to find causal relationships between the model's internal representations and its downstream bias, we could more effectively measure and intervene on these undesirable behaviors. we therefore turn to a recent series of papers on interpretability methods that focus on causal discovery. in section 2 we discuss three such methods, of which we adapt one (diffmask) for our needs in section 3. our new method is more efficient than other causal methods, which is especially relevant when applied to large language models (llms). in section 3 we also report results from these three methods when applied to gpt2-small and the problem of gender bias, and find that they discover largely overlapping sets of components, despite huge differences in computation requirements. in section 4 we use the identified components to adapt gpt-2 small, using parameter-efficient fine-tuning procedures. we demonstrate how gender bias in lms can be reduced with minimal effect to their language modelling performance by making targeted interventions to their components. however, we also recognize the limitations of operationalizing gender bias as we do, using minimal pairs of contrasting sentences-which simplify gender as a binary construct and may not work so well for other languages than english-and call for future research to develop reliable and validated bias measures (see @xcite . ## related work where and how lms implement output behaviorsfrom high-level phenomena like gender stereotypes, to lower-level ones like subject-verb agreement-is an active field of study. in providing an overview of related work, we focus on causal methods for locating mechanisms in section 2.1, as non-causal methods can yield misleading conclusions @xcite . further, we review previous work on targeted changes to language models and their behavior in section 2.2 ## locating gender bias in this section, we investigate the question: where in a given lm is gender bias introduced? we study this in gpt-2 small @xcite , an english-language, auto-regressive pre-trained transformer lm. foot_0 its small size-12 transformer layers, with 12 attention heads and 1 multi-layer perceptron (mlp) each-makes it a good object of close studies like we perform. we seek to identify the subset of the 144 attention heads that introduce gender bias into the last position of gpt-2's input, where gpt-2 produces next-token predictions. we identify these heads in the context of inputs that lead to gender-biased next-tokens from gpt-2. this study thus focuses on attention heads. though prior work has emphasized the role of mlps in gender bias and memorization @xcite @xcite , we argue that attention heads are also an interesting subject of analysis. unless the final word of the input contains gender information that causes the production of biased next-tokens, this information must be introduced from other positions via attention heads. to determine where gpt-2 small introduces gender bias into its output, we use three methods: causal mediation analysis (cma), automated circuit discovery, and our own novel method that combines the first approach with differential masking. we then compare the results of these three methods. ## mitigating gender bias having identified components responsible for gender bias in gpt-2 small, we test whether this information can be used to mitigate the bias. to this end, we fine-tune the model on a dataset carefully curated to be gender balanced-this has been shown to lead to a reduction in gender bias @xcite . we compare the effectiveness of fine-tuning only the components found in the previous section to various baselines, both fine-tuned and not. ## discussion & conclusions with this work, we provide an exploratory study of the identification and mitigation of gender bias in gpt-2. our three different methods identify model components relevant to gender bias-according to our results, they largely agree on the most relevant attention heads: most of the heads responsible for gender bias are found mainly in the last four attention layers. we then intervene on each method's found components to mitigate the gender bias but maintain language modeling performance. we find that language modeling performance deteriorates only minimally for our 'narrow' interventions, but deteriorates more in conditions where a larger amount of components/parameters are adapted by fine-tuning. regarding computational efficiency, we find that the circuits approach is computationally inefficient compared to the other methods. for explanatory and exploratory work, like ours, circuits are very useful and can yield fine-grained insights into the model mechanisms. however, if resource efficiency is a high priority, we suggest using other methods than (automatic) circuit discovery. one key contribution of this paper is a new and very efficient method, diffmask+, which finds a minimal set of attention heads for fine-tuning, while being computationally less prohibitive than methods such as automatic circuit discovery. limitations have we reached our goal of reducing bias, using computational efficient methods? considering the measured gender bias, we successfully reduced the bias on two out of three datasets. this is encouraging, but our results also reveal some inconsistencies between different ways of measuring bias. this is not unexpected; in fact, much previous work has highlighted many issues that put the validity and reliability of current bias measures into question (e.g., @xcite @xcite . bias measures may target very different manifestations of the bias of interest @xcite . we therefore attribute the observed inconsistencies to the implicit versus explicit gender bias in different datasets, which could be represented differently in model components, and thus also targeted differently by fine-tuning. despite these challenges, we tried to address some of these concerns by using multiple different bias metrics and testing the consistency of these across different seeds. we believe that the success of our approach is heavily contingent upon the datasets employed for both component identification and the subsequent fine-tuning of the chosen components. for example, using template-based datasets such as winobias or professions could reduce the identified components' generalizability, as components that contribute to one form of gender bias may not contribute to another. the same applies to the fine-tuning stage as well. using a dataset with limited variability in structure might result in only partial mitigation of the behavior. we therefore conclude that for even better bias reduction, it is essential to use and develop datasets that are diverse and representative of the behaviour being studied. future work for a wider picture of how our findings integrate in bias identification and mitigation studies, we would like to compare our approaches to other promising methods in the literature like concept erasure at the activation level (e.g., leace; @xcite and changes to the language generation procedure (e.g., "selfdebiasing"; @xcite . future work should also test whether these mitigation strategies generalize to different conditions, for example, language models larger than gpt-2 small. lastly, we also stress the importance of developing methodologies for operationalizing other forms of bias than binary gender in english, and to overcome difficulties we currently face when using contrastive sets and existing bias benchmarks.