Towards Understanding Task-agnostic Debiasing Through the Lenses of Intrinsic Bias and Forgetfulness (2024)

Abstract: While task-agnostic debiasing provides notable generalizability and reduced reliance on downstream data, its impact on language modeling ability and the risk of relearning social biases from downstream task-specific data remain as the two most significant challenges when debiasing Pretrained Language Models (PLMs). The impact on language modeling ability can be alleviated given a high-quality and long-contextualized debiasing corpus, but there remains a deficiency in understanding the specifics of relearning biases. We empirically ascertain that the effectiveness of task-agnostic debiasing hinges on the quantitative bias level of both the task-specific data used for downstream applications and the debiased model. We empirically show that the lower bound of the bias level of the downstream fine-tuned model can be approximated by the bias level of the debiased model, in most practical cases. To gain more in-depth understanding about how the parameters of PLMs change during fine-tuning due to the forgetting issue of PLMs, we propose a novel framework which can Propagate Socially-fair Debiasing to Downstream Fine-tuning, ProSocialTuning. Our proposed framework can push the fine-tuned model to approach the bias lower bound during downstream fine-tuning, indicating that the ineffectiveness of debiasing can be alleviated by overcoming the forgetting issue through regularizing successfully debiased attention heads based on the PLMs’ bias levels from stages of pretraining and debiasing.

## introduction social fairness of plms has recently drawn intense critical attention, particularly due to the widespread deployment of plm-based systems @xcite @xcite . social biases embedded in plms can drive plm-based systems to generate stereotypical content with respect to underrepresented demographic groups, raising serious issues of social fairness @xcite . therefore the process of debiasing plms to better align them with social values of fairness is a key procedure before deploying plms for public access @xcite . to illustrate the unintended behavior of social bias, a popular example is: the surgeon asked the nurse a question, he ...; the nurse asked the surgeon a question, she .... given the occupation token, surgeon, in the context of "the surgeon asked the nurse a question", plms are more likely to make a generation decision to assign the binary gender token he, instead of she, by referring to the occupational token. this indicates that plms predict surgeons as male with a higher probability than surgeons as female, presenting an example of gender bias @xcite . intrinsically, plms amplify the statistical bias in the pretraining corpus where the concurrence between surgeon and he is much larger than that between surgeon and she @xcite . despite various studies highlighting social bias issues @xcite @xcite , the effectiveness of debiasing for downstream applications continues to be debated @xcite @xcite . when it comes to debiasing, the language modeling abilities @xcite and relearning of social biases @xcite are the two main concerns limiting the effectiveness of debiasing. considering counterfactual data augmentation (cda) @xcite as an instance of debiasing, the lower quality of the debiasing corpora compared to the pretraining corpora negatively impacts the language modeling ability, therefore degrading downstream performance. earlier studies have arrived at varying conclusions regarding the effectiveness of debiasing in reducing social bias in fine-tuned tasks. @xcite and @xcite claim that a debiased model can help with downstream tasks, but @xcite empirically demonstrates that fine-tuning a debiased model for downstream tasks can lead to significantly biased models @xcite . however, an in-depth understanding of this ineffectiveness is still under-studied. this paper focuses on the relearning of social bias challenge and proposes a framework to alleviate this problem via an in-depth understanding of how plms' parameters change during debiasing and fine-tuning. we empirically indicate that debiased plms are sensitive to bias in downstream data through a comprehensive analysis of the bias score of the fine-tuned model given various bias levels foot_2 in downstream data. our observations indicate that: (1) the bias level of the debiased plms is the approximate lower bound for any fine-tuned plms for practical cases, and (2) relearning social biases derives from the forgetting issue of plms @xcite . when fine-tuning occurs in downstream tasks exhibiting higher bias levels, the resultant model tends to display greater bias compared to the initial debiased model. through meticulous control of bias levels within downstream tasks, we can conclude that the effectiveness of task-agnostic debiasing is dependent on the bias level of both the debiased plms and the downstream data. to thoroughly understand how the attention heads of a plm change, and how those changes are associated with social biases and downstream generalization, we propose prosocialtuning. specifically, we implement a generalization importance estimation method based on pac-bayes training, which indicates parameters' importance by learning parameter-wise noise variance through minimizing a variant of a pac-bayes bound in a posttraining manner @xcite . a higher noise variance indicates less importance to generalization. in the downstream finetuning stage, we apply regularization to successfully debiased attention heads, guided by their importance to downstream generalization. in section 2 we introduce relevant works. section 3 introduces our first main contribution: the use of the bias level as an approximate lower bound. section 4 presents the necessary mathematical and algorithmic background context for our second main contribution: our novel framework, proso-cialtuning. the remaining sections detail proso-cialtuning and its experimental evaluation. our contributions are threefold: (1) we provide an empirical resolution to the debate regarding the effectiveness of task-agnostic debiasing during downstream fine-tuning, specifically in the context of relearning social bias; (2) we elucidate the underlying principle of the relearning social bias issue; and (3) we propose a novel solution to address this issue. ## related works the effectiveness of a separate step of debiasing before downstream fine-tuning has been explored in recent studies. @xcite implemented comprehensive studies on the intrinsic bias of plms and extrinsic bias of fine-tuned plms in downstream applications, in terms of gender bias. recently, lalor et al. (2024) proposed a modelbased evaluation metric for social bias evaluation. their experimental results showed that a debiasing step is less effective for downstream tasks, contrary to the conclusion of debiasing transferability in @xcite . @xcite indicates the intrinsic bias evaluation metric is not correlated to application bias. a similar conclusion is presented in @xcite , in which the authors investigate the bias transfer hypothesis and prove that debiasing cannot help mitigate bias in fine-tuned tasks. @xcite proposed causal-debias to solve the ineffectiveness of debiasing but their assumption about causal factors is too strong and cannot generalize to other datasets well. pac-bayes training is a training algorithm which differs from conventional empirical risk minimization in that it optimizes a machine learning model by minimizing a generalization error bound (the pac-bayes bound). @xcite ing context of high dimensional parameters and a small training dataset size. pac-tuning is an extension of @xcite , which introduced a pac-bayes training method that optimizes both the prior and posterior variance of the model's parameters, and proposed a new pac-bayes bound for unbounded classification loss. ## bias lower bound in this section, we present the first major contribution of this work: that the bias level, i.e., the level of a specific type of bias (e.g., gender bias) of a debiased model can be leveraged as an approximate lower bound for optimizing the fine-tuning of plms, given a biased fine-tuning dataset. with this, we aim to close the debate about the ineffectiveness of debiasing via experiments highlighting extreme cases. we began by investigating the correlation between the effectiveness of debiasing and the bias levels in the debiased model and downstream tasks, in the context of the gender bias task. to do so, for different datasets, we compare the bias score of fine-tuned models, as measured by the stereoset score foot_3 , with respect to: (1) proportions of female gender-relevant samples, as defined by the gender word list in @xcite , and (2) dataset sizes, as shown in figure 1 . given a debiased model, we manipulate the bias levels in the training set and report the bias score of the fine-tuned model with respect to various bias levels. we use three datasets for analysis: multinli @xcite ) from the glue benchmark, the jigsaw unintended bias in toxicity classification foot_4 , and the stanford natural language inference (snli) corpus @xcite . to experiment with dataset sizes, we randomly sample data from the training dataset wherein no sentences contain femalerelevant words. we consider varying dataset sizes of 100, 500, 1000, 5000, and 10000 instances to analyze the impact of different training dataset sizes. to vary the bias levels with respect to genderrelevant samples across plms, we rebalance samples containing words relevant to the female gender in our training dataset. then we construct a training dataset with 10,000 samples and change the amount of samples with the pre-defined femalerelevant words. in our experiments, we systematically varied the proportion of sentences containing female gender words, setting it at 0.0, 0.25, 0.5, 0.75, and 1.0. subsequently, we calculated the average bias score across three different seeds for each of these proportion settings. to validate the effects of debiasing on the language modeling ability, we conducted experiments to gauge the language modeling score foot_5 . as shown in appendix figure 3 , the pearson product-moment correlation coefficients between the bias score and the language modeling score is less than 1. thus, we can focus on the effects of the bias levels of the data and mod-els, as those are the most straightforward factors in practical scenarios. according to figure 1 , the fine-tuned model indicates more bias than the debiased one in most cases, implying the ineffectiveness of debiasing. this is further verified by the lower bias score of the fine-tuned model versus the pretrained model (figure 1(b)-(c) ). these findings indicate that the bias level in the downstream task is less than that of the pretrained model. changing the bias levels in training data results in varying fluctuations of bias scores among fine-tuned models across the three evaluated benchmark tasks. the bias score gap between the fine-tuned model based on the pretrained model versus the debiased model is attributed to the disparity of their language modeling abilities. given the experimental results regarding varying dataset sizes (figure 1(d)-(f )), it is obvious that fewer training samples result in lower bias scores. therefore we can conclude that the bias levels of the downstream tasks are highly relevant to the debiasing effectiveness. remarkably, debiased + fine-tuned displays the highest bias scores (around 55) across various bias levels and tasks. conversely, fine-tuned has a peak bias score closely aligned with the bias score of the pretrained model. moreover, the lowest bias scores exhibited by debiased + fine-tuned with differing dataset sizes are strikingly akin to the bias score of the debiased model. however, the bias score of debiased + fine-tuned should be higher than the debiased model, considering downstream tasks are generally rather biased in practical scenarios. consequently, the efficacy of task-agnostic debiasing hinges upon both the bias level present in the downstream task data and the debiased model. the debiased model sets a definitive lower bound for the bias levels of the fine-tuned model after debiasing, as long as social bias exists within the downstream task data @xcite . inspired by this conclusion, in section 5, we prove that we can approach the lower bound of the bias level by regularization over the debiased model itself, without any additional debiasing methods or annotated datasets, given highly biased downstream tasks. ## background in this section, we present the mathematical and algorithmic context necessary for understanding our prosocialtuning framework. assume a plm f , consisting of l layers and k attention heads per layer, is parameterized by θ with attention weights as θ a . the k th attention head in the l th layer a l,k is parameterized by θ a l,k . we denote cma(f, d cma ) as the causal mediation analysis to the attention heads of f with dataset d cma , and denote cda(f, d cda ) as debiasing of plm f with the counterfactual data augmentation dataset d cda . for each training sample x i and its label y i , we denote the cross-entropy loss as l(x i , y i ; θ). ## prosocialtuning using the analysis of section 3 and bias-inducing attention shift (section 4.1), prosocialtuning shows that we can propagate debiasing efforts to downstream fine-tuning by only remembering the successfully debiased attention heads. this framework offers insight into understanding the resurgence of social bias in downstream applications. ## experiments in this section, we introduce the experimental settings and results of prosocialtuning, which indicate that an inability to address the forgetting issue in plms limits the effectiveness of debiasing. ## discussions with this paper we would like to explore empirical observations which will lead to more insights for theoretical analysis about how plms learn social bias and how we can efficiently mitigate social bias. this goal is challenging and non-trivial, but the following is a brief theoretical analysis approached from the frameworks of statistical learning theory and natural language processing. assuming that the bias level is linearly dependent on the generalization performance, that there are obvious biases in the fine-tuning task, and that the debiased model has been properly debiased, we can leverage the pac-bayes bound for this theoretical analysis. for instance, in section 3.5 of @xcite , m is the number of fine-tuning task samples. if there are more samples (larger m) in the fine-tuning, the bias level of the fine-tuned model should be relevant to the fine-tuning dataset size. the conclusion above is intuitive. however, the generalization behavior of llms is rather different from traditional machine learning models. for example, when using double-descent @xcite , or how the catastrophic forgetting issue seems to be less strong in very large llms @xcite ), yet generalization is still good. we believe extending prosocialtuning to much larger models will be helpful in terms of understanding task-agnostic debiasing. in this paper, we only focused on text classification tasks, wherein masked language models with fewer parameters are much more popular. besides our hardware limitations, we also have other reasons for this: (i) people tend to use instructions to leverage models with over several billions of parameters and there is no downstream fine-tuning, so the relearn-ing of bias issue as we study it does not hold; (ii) we observe a serious decrease in language modeling ability with cda and safety alignment, e.g., reinforcement learning from human feedback, can preserve the language modeling ability. however, the recently proposed superficial alignment hypothesis might indicate the ineffectiveness of this alignment method. regarding the bias lower bound, our claim is an empirical lower bound but not an exactly theoretical lower bound which requires more effort, although we tend to leverage empirical evidence to inspire future studies. learning and mitigating social bias is a system-level research topic, hindering the straightforward application of existing theoretical tools. for the empirical lower bound, we aim to analyze how the data influences relearning social bias and explore the role of the model with prosocialtuning. from the data/task perspective, the settings chosen for dataset size and ratio of female-relevant samples are the two most practical ones we can manipulate to study. ## future work and conclusion based on our findings, we anticipate that future research will: (1) propose theoretical proofs to validate the effectiveness of task-agnostic debiasing; this work addresses the ongoing debate surrounding the effectiveness of task-agnostic debiasing techniques for downstream tasks. our research reveals a pivotal factor determining the effectiveness of debiasing: the joint effect of bias levels of the debiased model and the downstream task dataset. specifically, the bias level of the debiased model serves as the approximate lower bound for bias in fine-tuned tasks wherein social bias exists. to gain an in-depth understanding of how forgetting changes plms' parameters, we introduce prosocialtuning, a novel framework that mitigates the diminishing effectiveness by imposing regularization on attention heads that have already undergone successful debiasing.