Unlearning Bias in Language Models by Partitioning Gradients (2023)

Abstract: Recent research has shown that large-scale pretrained language models, specifically transformers, tend to exhibit issues relating to racism, sexism, religion bias, and toxicity in general. Unfortunately, these pretrained language models are used almost universally in downstream tasks, and natural language processing is often applied to make real-world predictions. Thus, debiasing these language models as early in development as possible is increasingly crucial for preventing unintentional harms caused by natural language systems. To this end, we propose a new technique called partitioned contrastive gradient unlearning (PCGU), a gray-box method for debiasing pretrained masked language models. PCGU aims to optimize only the weights that contribute most to a specific domain of bias, doing so by computing a first-order approximation based on the gradients of contrastive sentence pairs. Our experiments show that PCGU is both low-cost and seems particularly effective at pinpointing the sources of implicit social bias in large pretrained transformers. Although we train using PCGU in the gender-profession domain only, we find that doing so can also partially mitigate bias across other domains. All code for our implementation and experiments can be found at https://github.com/CharlesYu2000/PCGU-UnlearningBias .

## introduction in the past few years, extraordinary improvements have been made to most applications of natural language processing due to the prevalence of large pretrained language models, particularly transformers @xcite . these language models achieve remarkable performance not only because of mechanisms like attention @xcite , but because of rich and diverse natural language corpora scraped from literature and the internet. however, in spite of some measures to ensure that these natural language sentences are high quality @xcite , recent work has shown that pretraining corpora contain many toxic/biased sentences and that neural models trained on such data readily capture and exhibit these biases @xcite @xcite . previous studies suggest that embeddings and models encode harmful social biases @xcite @xcite @xcite @xcite . this can be problematic, as the lack of interpretability in modern language models means that negative stereotypes and social biases encoded in models may lead to unfairness and harms in production systems. without effective mitigation techniques, finetuned models utilizing these flawed language representations might accidentally inherit spurious correlations not representative of the real world or their target task. to mitigate the representational harms explained in @xcite ; @xcite , we might aim for two goals of different granularities. the first goal proposes to debias a model such that its predictions encode the least bias. the second aims to remove social bias throughout a model such that the model minimally represents constructs that can cause itself to be biased in its predictions. regardless of the debiasing goal, the north star is to eliminate harms caused by the model, so we must be motivated by how pretrained language models are used. minimizing the cost of adoption for debiased language models is a high priority for debiasing, as any barriers may cause people to be skeptical of the societal benefits. to ensure that people have little reason not to use our debiased model, we aim to minimize representing bias while still maximizing the representation ability of the model. in this study, we focus on debiasing pretrained language models used directly for masked language modeling. crucially, we modify only their weights post-hoc without any changes to the architecture or additional modules. in this way, we enable key stakeholders to swap out their masked language models (by simply loading a different set of weights) but still use the exact same code for masked predictions, just as they might with any other finetuned model. furthermore, stakeholders need not rely on the people pretraining the model to have incorporated debiasing procedures during the pretraining process. we restrict our study to masked language modeling, as the use cases of language models for other downstream tasks are disparate, and extrinsic evaluation of bias in those tasks is often be confounded by task-specific finetuning @xcite . we expect, based on the results from @xcite ; @xcite , that problematic social biases propagate throughout large portions of language models. furthermore, based on the lottery ticket hypothesis @xcite , we hypothesize that most bias is encoded by specific groups of neurons rather than individual weights throughout the model. so, we propose a gradient-based debiasing method called partitioned contrastive gradient unlearning (pcgu) to locate where in the model these problematic inferences originate from and to systematically retrain those parts of the model to unlearn this biased behavior. in our experiments, we use pcgu to unlearn biases in the gender-profession domain and evaluate our approach using prior association tests for bias/stereotypes. we find that pcgu is seemingly effective both in mitigating bias for the gender-profession domain that it is applied to as well as for generalizing these effects to other unseen domains. in addition, we observe that the procedure exhibits results quickly, requiring very few iterations over the tuning dataset and very little real time until convergence. the hyperparameter search space can be found in appendix a. ## related work motivated by the idea that the words in sentences are the root of all the information flowing through language models, static word embeddings were the first target for debiasing @xcite @xcite @xcite @xcite . these methods typically operate via projection onto some subspace that does not en-code the targeted bias. however, modern language models do not use external embeddings, so it is not immediately clear that such methods can be applied to transformers. further efforts have been made to extend those patterns for contextualized embeddings @xcite @xcite . however, such studies typically do not account for interactions between different parts of the model when used in actual sentences. instead, they focus either on the (static) word embedding layer or on aggregate representations of specific words. methods that propose debiasing models beyond the word level have also been proposed @xcite . however, most of these methods aim only to improve the case where another model will further use the sentence representations generated by the text encoder. crucially, this does not solve any word-level problems such as masked language modeling. for example, methods like @xcite add on extra modules, which means that the cost of adoption is more than simply loading a new weights file. in a different vein, methods like @xcite utilize multiple iterative prompts to debias generations only. recently, much work in this field has been focused on changing the pretraining or finetuning process to prevent bias from being learned by the language model. many approaches aim to change the training process for embeddings, classifiers, or encoders, either through changing the training procedure or adding bias-aware terms to the training loss function @xcite . some of this work has achieved success by attempting to "neutralize" the language models' representation of biased words over some bias subspace by finetuning @xcite or prompt tuning @xcite , or by extending these ideas by reformulating the bias dimensions as a set of implicit dimensions from social psychology @xcite . other methods propose changing or augmenting the training data in some way, typically by adding high-quality unbiased or antistereotypical sentences, eliminating blatantly biased or stereotypical sentences, or a combination of the two by replacing texts in the training corpus @xcite @xcite . yet other techniques utilize counterfactual or adversarial signals to dissuade models from encoding biases @xcite layer n layer 2 layer 1 x w k @xmath0. p(his) = 0.92 p(her) = 0.07 bert step 1: partition parameter set individual weight vectors step 2: find the weight vectors encoding bias get gradients bert backpropagate step 3: unlearn bias vec angle 180 176 153 92 32 12 least bias -most bias bias: non-bias: gradient descent to minimize advantaged probability for bias weight vectors choose top-k (@xmath1; @xcite @xcite @xcite . perhaps most similar to our method is actually work done in the knowledge editing space. such tasks propose explicitly editing specific knowledge in a model without affecting unrelated knowledge @xcite . this is quite similar to our task in that we aim to remove specific social bias from our model without affecting unrelated inference ability. however, our method attempts to remove generalized forms of these biases, as opposed to removing/changing the more targeted and specific knowledge that knowledge editing methods attempts to do. recent studies include gradient-based methods that train separate networks to predict efficient gradient updates for removing or replacing models' knowledge @xcite . ## methods at a high-level, pcgu is composed of three parts. first, gradients must be computed for a contrasting pair of sentences whose difference is in the domain that the model is biased in. next, we apply a weight importance algorithm, based on gradients, to compute a ranked ordering of weights that are most important to our criterion (i.e., the weights that seem to most encode the biases we wish to unlearn). finally, taking the earlier gradients and ordered weights as input, we compute a first-order approximation of the bias gradient and perform a standard optimization step of our language model. in our experiments, we apply this procedure to debias a group of masked transformer language models for the gender-profession domain such that their final parameters encode less inequality for mlm predictions. specifically, we aim to update the models such that they are not generally biased toward a stereotypical sentence nor an antistereotypical sentence, since even antistereotypes can be harmful @xcite . we later evaluate pcgu's efficacy using existing evaluation benchmarks. ## conclusion in this paper, we introduced pcgu, a method to systematically search through a pretrained masked language model to find the origins of its bias and mitigate them. the positive results in our paper suggest that, with the proper data, post-hoc removal of problematic social biases can be efficient and targeted via pcgu. our findings also support the notion that different types of bias arise from different areas in pretrained transformers. we believe that by focusing on the language model holistically, rather than as a collection of individual pieces, we can more effectively remove representational harms from pretrained language models. it is our hope that future studies are able to leverage pcgu to fully debias language models and increase adoption of fair pretrained models.