What changed? Investigating Debiasing Methods using Causal Mediation Analysis (2022)

Abstract: Previous work has examined how debiasing language models affect downstream tasks, specifically, how debiasing techniques influence task performance and whether debiased models also make impartial predictions in downstream tasks or not. However, what we don’t understand well yet is why debiasing methods have varying impacts on downstream tasks and how debiasing techniques affect internal components of language models, i.e., neurons, layers, and attentions. In this paper, we decompose the internal mechanisms of debiasing language models with respect to gender by applying causal mediation analysis to understand the influence of debiasing methods on toxicity detection as a downstream task. Our findings suggest a need to test the effectiveness of debiasing methods with different bias metrics, and to focus on changes in the behavior of certain components of the models, e.g.,first two layers of language models, and attention heads.

## introduction recent work has shown that pre-trained language models encode social biases prevalent in the data they are trained on @xcite @xcite . in response to that, solutions to mitigate these biases have been developed @xcite @xcite . some recent papers also examined the impact of debiasing methods, e.g., reduction of gender bias, on the performance of downstream tasks, e.g., classification. @xcite @xcite . for example, @xcite showed that debiasing techniques worsened gender bias of a downstream classifier for occupation prediction. @xcite investigated how debiasing methods affect the model's language modeling ability. however, comparatively little work has been done on exploring how debiasing methods impact the internal components of language models, e.g., the models neurons, layers, and attention heads, and what kind of changes in language models are introduced when debiasing methods are applied to downstream tasks. in this paper, we apply causal mediation analysis, which investigates the information flow in language models @xcite , to scrutinize the internal mechanisms of mitigating gender debiasing methods and their effects on toxicity analysis as a downstream task. we first examine the efficacy of debiasing methods, namely, cda and dropout @xcite , on 1) language models, namely, bert @xcite and gpt2 @xcite , and 2) models (jigsaw, and rtgender) @xcite fine-tuned for downstream tasks. the debiasing methods (cda and dropout) were chosen because they had been shown to minimize detrimental correlations in language models while maintaining strong accuracy @xcite . we then applied causal mediation analysis to understand how internal components of a model are impacted by debiasing methods and fine-tuning. in this study, we focus on gender bias as a type of bias. we examine (1) stereotypical associations between gender and professions in pre-trained language models (seat) @xcite , (2) stereotypes encoded in language models (crows-pairs) @xcite , and (3) differences in systems affecting users unequally based on gender (wino-bias) @xcite . these representational harms can impact people negatively because they contribute to exacerbating stereotypes inherent in society. these harms may also result in unfavorable consequences when these language models are deployed for practical purposes, e.g., when a model behaves disproportionately against certain demographics @xcite . ## experimental setup models the experiment was conducted on two pre-trained language models: gpt2 (small) (radford et al., 2019) and bert (bert-base-uncased) @xcite . the configuration of the debiasing models is detailed below. cda wikitext-2 @xcite , and the gendered word pairs 1 proposed by @xcite is used in the pre-training phase. ## conclusion in this work, we have investigated how debiasing methods impact language models, along with the downstream tasks. we found that (1) debiasing methods are robust after fine-tuning on downstream tasks. in fact, after the fine-tuning, the debiasing effects strengthened. however, this effect is not supported across another bias measure. this indicates the need for both debiasing techniques and bias benchmarks to ensure generalizability. the causal mediation analysis suggests that (2) the neurons that showed a large change in behavior were located in the first two layers of language models (including the word embedding layers). this suggests that careful inspection of certain components of the language models is recommended when applying debiasing methods. (3) applying debiasing and fine-tuning methods to language models changes the weight that attention heads assign to gender-associated terms. this indicates that attention heads may play a crucial role in representing gender bias in language models. several limitations apply to this work. we only tested these effects on one downstream task, namely, toxicity detection. in order to check the generalizability of these findings, experiments with other downstream tasks are necessary.