Gender Biases and Where to Find Them: Exploring Gender Bias in Pre-Trained Transformer-based Language Models Using Movement Pruning (2022)

Abstract: Language model debiasing has emerged as an important field of study in the NLP community. Numerous debiasing techniques were proposed, but bias ablation remains an unaddressed issue. We demonstrate a novel framework for inspecting bias in pre-trained transformer-based language models via movement pruning. Given a model and a debiasing objective, our framework finds a subset of the model containing less bias than the original model. We implement our framework by pruning the model while fine-tuning it on the debasing objective. Optimized are only the pruning scores – parameters coupled with the model’s weights that act as gates. We experiment with pruning attention heads, an important building block of transformers: we prune square blocks, as well as establish a new way of pruning the entire heads. Lastly, we demonstrate the usage of our framework using gender bias, and based on our findings, we propose an improvement to an existing debiasing method. Additionally, we re-discover a bias-performance trade-off: the better the model performs, the more bias it contains.

## introduction where in language models (lm) is bias stored? can a neural architecture itself impose a bias? there is no consensus on this matter. @xcite suggest that gender bias resides on every layer of transformer-based lms. however, this is somehow vague -transformer layers can be further decomposed into building blocks, namely attention heads, and these also can be further broken down into matrices. on the other hand, the findings of @xcite show that some attention heads within layers specialize in particular tasks, such as syntactic and positional dependencies. this gives us an intuition that some heads, or their parts, may specialize in learning biases as well. being able to analyze bias in language models on a more granular level, would bring us a better understanding of the models and the phenomenon of bias. with knowledge of where the bias is stored, we could design debiasing techniques that target particular parts of the model, making the debiasing more accurate and efficient. we demonstrate a novel framework that utilizes movement pruning @xcite to inspect biases in language models. movement pruning was originally used to compress neural models and make its inference faster. we introduce a modification of movement pruning that enables us to choose a low-bias subset of a given model, or equivalently, find these model's weights whose removal leads to convergence of an arbitrary debiasing objective. specifically, we freeze neural weights of the model and optimize only the so-called pruning scores that are coupled with the weights and act as gates. this way, we can inspect which building blocks of the transformers, i.e. attention heads, might induce bias. if a head is pruned and the debiasing objective converges, then we hypothesize that the head must have contained bias. we demonstrate the utility of our framework using @xcite 's method of removing gender bias. biases have been extensively studied and numerous debiasing methods were proposed. in fact, according to @xcite , the acl anthology saw an exponential growth of biasrelated publications in the past decade -and it only counts gender bias alone. nonetheless, the vast majority of these works address problems of bias detection or mitigation only. to our best knowledge, we are the first to conduct bias ablation in lms. we: (1) demonstrate an original framework to inspect biases in lms. its novelty is a mixture of movement pruning, weight freezing and debiasing; ## exploring gender bias using movement pruning we focus on gender bias defined as stereotypical associations between male and female entities. our study is limited to the english language and binary gender only. we attempt to answer the following questions: in transformer-based pre-trained language models, can we identify particular layers or neighboring regions that are in charge of biases? to verify this, we propose a simple and, to our best knowledge, novel framework based on debiasing and attention head block movement pruning. given a pre-trained model and a fine-tuning objective, we find which attention blocks can be disabled, so the model performs well on the task. we prune the model while fine-tuning it on a debiasing objective, such as the one described in §2.1. we optimize solely the pruning scores s and the weights w of the original model remain untouched (they are frozen). we target the building blocks of transformerbased models, attention heads @xcite . each head consists of four learnable matrices, and we prune all of them. in §3.1, we test two strategies: pruning square blocks of the matrices and pruning entire attention heads. to evaluate bias, we utilize sentence encoder association test (seat, @xcite and stereoset stereotype score (ss, @xcite evaluated on the gender domain. to measure model performance, we utilize glue @xcite , a standard nlp benchmark. ## debiasing early intermediate layers is competitive kaneko and bollegala (2021) proposed three heuristics: debiasing the first, last, and all layers. however, the number of layer subsets that can be debiased is much larger. trying all subsets to find the best one is prohibitively expensive. with our framework, we are able to find a better subset with a low computational cost. we observed that: (1) square block pruning does not significantly affect the first and last layer: densities of these layers are usually higher than the other layers' (fig. 1 ); (2) attention head pruning mostly affects intermediate layers (fig. 2 ). based on the above, we propose to debias intermediate layers. specifically, we take the embeddings from layers index 1 to 4 inclusive, and we run the debiasing algorithm described in §2.1. we do not include layer 0 because it generally yields high densities (ref. fig. 1 ), and layer 5, as it contains the most number of heads that were not pruned in every experiment (ref. fig. 2 ). we end up with two more modes, intermediate-token and intermediate-sentence. we present results for our, as well as the other modes in tab. 4 (note that the results may differ from @xcite 's due to random seed choice). debiasing the intermediate layers is competitive to debiasing all and last layers. the ss of the intermediate-modes is lower that the ss of corresponding all and last modes. the ss of intermediate-sentence gets close to the perfect score of 50. ## conclusion we demonstrate a novel framework to inspect sources of biases in a pre-trained transformer-based language model. given a model and a debiasing objective, the framework utilizes movement pruning to find a subset that contains less bias than the original model. we present usage of our framework using gender bias, and we found that the we hope that in the future our framework will find more applications, not only limited to gender bias. provided by @xcite , and a list of stereotyped targets provided by @xcite .