B ias E dit: Debiasing Stereotyped Language Models via Model Editing (2025)

Abstract: Previous studies have established that language models manifest stereotyped biases. Existing debiasing strategies, such as retraining a model with counterfactual data, representation projection, and prompting often fail to efficiently eliminate bias or directly alter the models’ biased internal representations. To address these issues, we propose BiasEdit, an efficient model editing method to remove stereotypical bias from language models through lightweight networks that act as editors to generate parameter updates. BiasEdit employs a *debiasing loss* guiding editor networks to conduct local edits on partial parameters of a language model for debiasing while preserving the language modeling abilities during editing through a *retention loss*. Experiments on StereoSet and Crows-Pairs demonstrate the effectiveness, efficiency, and robustness of BiasEdit in eliminating bias compared to tangental debiasing baselines, and little to no impact on the language models’ general capabilities. In addition, we conduct bias tracing to probe bias in various modules and explore bias editing impacts on different components of language models.

## introduction in recent years, many studies have underscored the tendency of pre-trained language models (lms) to have societally stereotypical biases @xcite @xcite , such as gender bias @xcite , race bias @xcite , religion bias @xcite , among others. therefore, eliminating biases from models is crucial to ensure fairness and accuracy in applications of language models. many methods have been proposed to mitigate bias, such as fine-tuning entire models @xcite with counterfactual data obtained by swapping out bias attribute words, foot_1 which is partly effective but costly in terms of computational time and space, especially for large language models (llms). others implement debiasing with representation projection @xcite @xcite or prompting @xcite @xcite . however, without parameter modification, a model remains inherently biased and can not be applied to downstream tasks as an off-the-shelf unbiased model. recent methods @xcite employ model adapters where each adapter is trained to specialize only in one bias type. multiple adapter training for different bias types is not economical for real-world applications. these drawbacks inspire us to explore new methods for debiasing stereotyped language models more directly. model editing @xcite @xcite can change specific information in language models by modifying model parameters, which could be effective in eliminating bias. there are some existing edit-ing methods: (i) fine-tuning a model with new data @xcite ; (ii) locating then editing @xcite @xcite @xcite ; (iii) utilizing editor hyper-networks to modify language models' parameters @xcite @xcite . as for current llms (@xmath0, the fine-tuning approach consumes a lot of computational resources and data, which is not ideal. recent works @xcite @xcite and our preliminary experiments (see appendix a) show that bias can be interpreted as localized modules in llms. meanwhile, small hyper-networks predicting weight updates @xcite @xcite are illustrated to be flexibly applied to change parameters of any language models without fully fine-tuning it and adaptively designed to conduct any specific editing task. in §3, therefore, we introduce biasedit, a lightweight model editing approach to debias stereotyped language models using editor hypernetworks, as illustrated in figure 1 . biasedit aims to calibrate a language model's biased behavior to assign the same likelihoods to the stereotyped contexts and their corresponding anti-stereotyped contexts. inspired by @xcite and @xcite , biasedit uses editor networks to modify a small portion of model parameters relating to stereotyped bias and then obtain an off-theshelf unbiased model for downstream applications. a debiasing loss in biasedit is designed to teach editor networks how to generate parameter shifts to modify partial parameters of language models for debiasing. biasedit also contains a retention loss to avoid affecting unrelated associations during editing to preserve language modeling abilities. to demonstrate the effectiveness and robustness of bi-asedit, we conduct experiments on the stereoset @xcite and crows-pairs @xcite datasets with four different lms compared to previous debiasing methods. the results show that biasedit achieves the best performance on debiasing than all baselines and has little impact on lms' language modeling and general abilities ( §4.2). meanwhile, biasedit is robust to gender reversal ( §4.5) and semantic generality ( §4.6). furthermore, we explore bias associations among various modules and the process of debiasing via model editing on different components of language models. we find that bias editing on up-per blocks of language models has fewer negative impacts on language modeling abilities than editing on the bottom blocks, shedding light on future debiasing research. ## biasedit to conduct effective and efficient debiasing, we propose biasedit, a model editing method for debiasing stereotyped language models. according to §2.2, given a language model with parameters θ, bias editing can be denoted as a function x × l × θ × φ → θ, which maps a paired input (x stereo , x anti ), a debiasing loss function l d : x × θ → r, biased language model parameters θ, and editor parameters ϕ to new unbiased model parameters θ e . as shown in figure 2 , biasedit utilizes lightweight networks as editors ϕ to generate a parameter shift, which is used to modify models' partial weights w (e.g., the weights of the last linear layer in the mlps at the last 3 blocks) for conducting debiasing edits, following the architecture of mend @xcite and malmen @xcite . specifically, (x stereo , x anti ) is used to compute the input to an editor network g ϕ ℓ for the layer ℓ, the gradient ∇ w ℓ l d (x stereo , x anti , θ). the output of g ϕ ℓ is the parameter shift ∇w ℓ to update w ℓ into @xmath1. biasedit uses a debiasing training set s train edit and a development set s dev edit to learn editor parameters ϕ. during training, the debiasing loss l d teaches editor networks how to produce parameter shifts to change w for eliminating bias: overall, the total editing loss for training editor networks is l e (ϕ) = l d + λl r . for evaluation, bias editors produce debiasing edits on a test set s test edit . because the effectiveness of instance-editing that uses one instance in each editing operation is limited @xcite @xcite @xcite , biasedit adopts batch-editing, which uses one-batch samples in one edit for the debiasing scenario. during both training and testing, the same batch size is used for optimal debiasing performance. an unbiased model is expected to have a ss of 50%. as for language modeling and general capabilities, we use the language modeling score (lms) from stereoset. it is the percentage of samples in which a model ranks meaningful associations over meaningless associations. we compute the average ss and lms for pre-edit models and post-edit models (ss pre-avg , ss post-avg , lms pre-avg , lms post-avg ) of all batch edits. an ideal debiasing will not change the lms before and after debiasing. we report ss pre-avg , ss post-avg , and ∆@xmath2. dataset. we utilize two bias benchmark datasets, stereoset @xcite and crows-pairs @xcite . there are three reasons to choose them. first, stereoset and crows-pairs are widely used @xcite @xcite @xcite @xcite @xcite . in addition, they cover various types of bias in models, including gender, race, and religion bias, which are evaluated in our paper. moreover, the meaningless attribute terms in stereoset can be applied to retain language modeling abilities during debiasing. as for stereoset, we stochastically split in the test set (3,526 samples) of the intrasentence stereoset by 8:1 as s train edit and s dev edit respectively and use the development set (1,292 samples) as s test edit , where attribute terms in s train edit and s dev edit are disjoint from s test edit . crows-pairs is also used as s test edit to evaluate biasedit's debiasing performance (details in appendix b). we also select three large language model benchmark datasets, openbookqa @xcite , boolq @xcite , and copa @xcite , to evaluate lms' capabilities of reading comprehension, knowledge questionanswering, and commonsense reasoning, respectively. their evaluations are conducted by open-compass tool @xcite and measured by accuracy based on perplexity. comparison. compared with biasedit, four distinguishing baseline debiasing methods from @xcite are implemented foot_3 : counterfactual data augmentation (cda) @xcite , sentencedebias @xcite , self-debias @xcite , and iterative nullspace projection (inlp) @xcite (details in appendix b.3). unlike all baselines, our editor networks can be trained with a mixture of all three types of bias, instead of dealing with only one particular bias at a time. as for testing, bi-asedit is evaluated on gender, race, and religion bias samples from s test edit separately. biasedit is a model-agnostic debiasing method and can be applied to any open-sourced language model. we conduct experiments on diverse language models, including gpt2 @xcite , gemma @xcite , llama3 (meta, 2024), and mistral @xcite . some blocks in lms are selected in this paper according to preliminary ## related work bias and debiasing many works focus on measuring bias in language models @xcite @xcite @xcite @xcite , which provide bias measurement metrics @xcite . to mitigate bias, researchers propose various debiasing methods @xcite . the basic method is to fully finetune language models on counterfactual data @xcite , which is costly. so other approaches adopt fine-tuning in an efficient way @xcite @xcite . except for fine-tuning, prompting @xcite guides models to calibrate their bias. representation projection @xcite is employed to remove bias representation out of models, which, however, cannot change the language models' internal bias in essence without modifying parameters. some works @xcite construct an adapter for each type of bias and plug it into a lm. if we want to mitigate n types of bias, n adapters will be trained, which is not efficient. recently, an em-pirical study @xcite has explored the feasibility of debiasing via model editing. therefore, we adopt model editing by efficiently editing partial parameters for debiasing lms. model editing much factual knowledge is memorized in language models @xcite @xcite @xcite . as the real world develops, some facts become obsolete and different over time. it is necessary to change, add, or erase facts stored in existing pre-trained language models @xcite . model editing @xcite is come up with to modify information in plms. editing should follow some properties @xcite : reliability (predicting updated facts), locality, generality, and efficiency (efficient in runtime and memory). the direct but inefficient editing is to fully finetune a model on new facts @xcite . for locality, many works @xcite @xcite @xcite seek the model parameters strongly related to the facts and then edit these localized hidden states. with high efficiency, @xcite ; @xcite achieve fast editing by training specific editor networks. also, lifelong model editing, like wise @xcite , is paid attention to for practical applications. recently, model editing has been applied to unlearn information from language models @xcite @xcite . inspired by them, we propose an efficient bias editing method, biasedit, to eliminate bias in language models while preserving the language modeling capabilities and generalizing gender reversal inputs and semantically related inputs. ## conclusion we propose biasedit, an efficient model editing method to debias stereotyped language models by modifying a small portion of language models' parameters with small editor networks. we design a debiasing loss l d for debiasing and a retention loss l r to maintain the language modeling abilities during editing. experiments illustrate that biasedit presents much better debiasing performance than classical debiasing methods and gives little to no harmful impact on language modeling and general capabilities. also, biasedit is robust in gender reversal and semantic generality. meanwhile, we comprehensively investigate the effects of debias-ing different components of language models.