Mitigating Biases for Instruction-following Language Models via Bias Neurons Elimination (2024)

Abstract: Instruction-following language models often show undesirable biases. These undesirable biases may be accelerated in the real-world usage of language models, where a wide range of instructions is used through zero-shot example prompting. To solve this problem, we first define the bias neuron, which significantly affects biased outputs, and prove its existence empirically. Furthermore, we propose a novel and practical bias mitigation method, CRISPR, to eliminate bias neurons of language models in instruction-following settings. CRISPR automatically determines biased outputs and categorizes neurons that affect the biased outputs as bias neurons using an explainability method. Experimental results demonstrate the effectiveness of our method in mitigating biases under zero-shot instruction-following settings without losing the model’s task performance and existing knowledge. The experimental results reveal the generalizability of our method as it shows robustness under various instructions and datasets. Surprisingly, our method can mitigate the bias in language models by eliminating only a few neurons (at least three).

## introduction instruction-following language models perform various tasks using instruction-based prompts @xcite @xcite . however, language models have suffered from undesirable biases, failing to follow user instructions despite their significant competency @xcite @xcite . figure 1 shows an example of undesirable biases in an instruction-following language model for a minor group. in this case, a language model assigns a high probability to a minor group label for a negative question despite the ambiguity of figure 1 : the example contains an ambiguous context; thus, the correct answer is "can't answer" since it is impossible to judge which group corresponds to the answer for a given negative question (e.g., "what group of people uses drugs?"). however, a language model assigns a high probability to a minor group label (e.g., "poor people"). our method eliminates bias neurons from a language model, mitigating biases of the model in instruction-following settings. the given context. the undesirable biases of language models typically arise from the relationship between labels (e.g., "poor people") and tokens (e.g., "drugs") within data instances @xcite . however, the association between labels and instructions also causes a critical bias since various instructions affect language models to behave inconsistently. figure 2 shows the inconsistent behavior of the flan-t5-base in various synonymous instructions on four datasets @xcite . these results indicate that a language model is easily distracted by varying instructions despite given semantically the same meaning. these phenomena suggest that language models exhibit significant cognitive biases in understanding figure 2 : performance gaps in understanding instructions. we plot the accuracies of flan-t5-base about ten synonymous instructions for bbq-ses, bbq-age, mrpc and rte datasets. for example, the accuracy between instructions differed by up to 5% and 15% for bbq-ses and mrpc, respectively. these results reveal that an instruction-following model shows biases in understanding instructions. the utilized instructions are described in detail in appendix d. to mitigate biases in language models, @xcite ; @xcite have investigated label biases in few-shot in-context learning settings. specifically, they have regarded the imbalanced probability distribution that occurred by inputting content-free texts (e.g., "n/a" or random tokens) as biases, and degraded the original output probability of each input instance by the output probability of the content-free texts. however, they have only aimed to mitigate biases in few-shot in-context learning settings, not considering the zero-shot instruction prompting. since utilizing a language model in only the instruction prompting without few-shot examples is an efficient and practical usage scenario of language models, they have significant limitations in the scope of application. to resolve these issues, in this paper, we first define the bias neuron, which significantly affects biased outputs in language models. prior studies @xcite @xcite have demonstrated that skills for a specific task are localized in particular neurons. from these findings, we hypothesize the presence of the bias neuron and empirically validate its existence. furthermore, we propose a novel bias mitigation method called crispr, which stands for calibrating instruction bias via bias neuron pruning. we utilize an attribution @xcite , an explainability method for language models, to quantify the bias of each neuron at three aspects (i.g., token, instance, and instruction). specifically, we first compute the bias influence of each token in inferring a predefined biased output, and aggregate the token score to an instance and instruction scores to derive the final bias influence (bias attribution) of each neuron. furthermore, defining biased outputs manually for each data instance is time-consuming; thus, we propose an automatic identification method of biased outputs for computing the bias attribution using the confusion score of a language model. we demonstrate our method in various social bias and natural language understanding benchmarks and dramatically outperform other baselines under varying zero-shot instruction settings. surprisingly, we reveal that only a few bias neurons (at least three) cause the bias, proving our method's practicality. in addition, we show that mitigating bias for a particular task does not adversely affect the existing knowledge of language models for solving other tasks. we also note that bias neurons identified for a specific dataset also function as biases in other analogous datasets, revealing that the bias knowledge is transferred to datasets from correlative domains. crispr is an efficient bias mitigation method since it needs only a few data samples (e.g., ten samples) to quantify the bias score for the whole neurons. crispr also enables language models to adapt flexibly by eliminating some existing bias neurons without any training process. ## methods in this section, we describe the process of quantifying and mitigating the bias of instruction-following language models. specifically, we compute attribution scores of each neuron for inferring automatically defined biased output. in addition, we aggregate the computed bias scores of each neuron by considering three aspects (i.g., token, instance, and instruction) to quantify biases in instructionfollowing settings. finally, we categorize bias neurons using the bias score and eliminate them using a pruning method to mitigate biases in language models. ## conclusion in this study, we define the bias neuron and prove its existence empirically. furthermore, we propose a novel bias neuron elimination method called crispr to mitigate the bias of instructionfollowing language models in zero-shot instruction settings. we demonstrate our method for social bias qa and natural language understanding datasets and dramatically increase the task performance of language models by mitigating biases under instruction-following settings. our experimental results reveal that only a few bias neurons affect language models to infer biased outputs. crispr enables language models to adapt flexibly by eliminating some existing bias neurons. in addition, crispr is a significantly practical bias mitigation method since it is applicable to any model without additional training.