Identifying and Mitigating Annotation Bias in Natural Language Understanding using Causal Mediation Analysis (2024)

Abstract: NLU models have achieved promising results on standard benchmarks. Despite state-of-the-art accuracy, analysis reveals that many models make predictions using annotation bias rather than the properties we intend the model to learn. Consequently, these models perform poorly on out-of-distribution datasets. Recent advances in bias mitigation show that annotation bias can be alleviated through fine-tuning debiasing objectives. In this paper, we apply causal mediation analysis to gauge how much each model component mediates annotation biases. Using the knowledge from the causal analysis, we improve the model’s robustness against annotation bias through two bias mitigation methods: causal-grounded masking and gradient unlearning. Causal analysis reveals that biases concentrated in specific components, even after employing other training-time debiasing techniques. Manipulating these components by masking out neurons’ activations or updating specific weight blocks both demonstrably improve robustness against annotation artifacts.

## introduction current natural language understanding (nlu) models obtain state-of-the-art accuracy on in-distribution benchmarks. however, researchers @xcite have found that these models utilize annotation bias to make predictions, negatively affecting the models' generalizability. consequently, bias analysis and mitigation are crucial topics in nlu. one popular approach to assessing bias in nlu models is behavioral analysis. this approach views models as test subjects and evaluates them by collecting their behavioral data in specific test cases without considering their internal components. results from behavioral analyses @xcite @xcite reveal that nlu models rely on superficial patterns to make predictions foot_0 . although using such superficial patterns as shortcuts can be advantageous when used in-distribution, they can mislead models to wrong answers when used in out-of-distribution settings. based on behavioral analysis, researchers develop techniques to address annotation artifacts directly by considering an entire nlu model as a black box. while this approach is beneficial in terms of simplicity, results from structural analysis show that learned features are localized within specific groups of components @xcite . consequently, we suggest that the study of bias mitigation would greatly benefit from analyzing an nlu model as a collection of components to direct the mitigation effort strategically. causal mediation analysis (cma) has been applied to conduct structural analysis on nlp tasks @xcite @xcite @xcite @xcite . cma provides a framework to identify specific components within a model that contribute to a behavior of interest. nevertheless, cma for nlu is underexplored, and common counterfactual generation techniques for cma rely on automatic text edits for each sample. this practice is inadequate to capture the spurious correlations we are interested in. furthermore, most cma works within nlp are limited to behavioral-structural analysis without using this valuable knowledge to enhance the performance of the models. to accommodate the need to interpret and mitigate biases, we argue that an ideal cma framework for nlu should have two following features: we apply the cma to analyze the nli models using the standard mnli dataset. in addition, our bias mitigation modules show improvements on the standard nlu spurious correlation mitigation benchmark that includes the following tasks: nli, fact verification, and paraphrase detection. cma shows that annotation bias effects are mainly concentrated in specific components, especially in neurons of the last few layers of the model. our experimental results on bias mitigation demonstrate that modifying components identified by the cma process can effectively improve the robustness against annotation artifacts. ## proposed method in this section, we formulate a method to identify model components prone to bias and mitigate bias in these components accordingly. as a structural analysis method, causal mediation analysis (cma) provides an effective means to analyze behavioral-structural relations within a pretrained language model (plm). in particular, this method has been used to investigate specific information in each plm component @xcite @xcite . however, no prior works explore how to manipulate the biascontaining components to mitigate spurious correlation in nlu tasks. the overview of our proposed framework is shown in figure 2 . sec. 3.1 presents our counterfactual generation method, enabling us to exploit known bias features to identify bias samples. sec. 3.2 describes the process that employed the generated counterfactuals to pinpoint components mediating biases. sec. 3.3 presents our bias mitigation methods that exploit the knowledge of these biased components to mitigate annotation biases. ## experimental results in this section, we show the results of causal mediation analysis (cma) on nlu tasks (section 5.1). to verify that the cma can pinpoint neurons that mediate annotation biases, we also provide bias mitigation results based on manipulating neurons identified by the cma as having high bias (section 5.2). moreover, we conduct ablation studies on the bias mitigation step to show that selecting neurons and "advantaged samples" based on the knowledge of annotation artifacts is an important factor in improving the robustness (section 5.3). in addition, we examine the effectiveness of mitigating the model-level effect of annotation artifacts on the prediction outcome (section 5.4). ## conclusion our investigation presents a novel approach to nlu bias mitigation research through causal mediation analysis (cma). in contrast to existing works that treat models as black boxes and primarily concentrate on loss engineering, cma performs component-wise analysis to pinpoint the neurons susceptible to bias. by employing cma, we have found that annotation bias effects are primarily concentrated within neurons located in the final layers of the model. through this newfound knowledge, we develop a strategy to direct our mitigation efforts to specific neurons and apply the two bias mitigation methods accordingly. experimental results show that we can reduce the impact of annotation artifacts through two bias mitigation methods. results from the ablation study confirm that cma is effective in identifying neurons that need to be rectified. by addressing the sensitivity of model modules to annotation bias, we pave the way for enhancing the robustness of nlu models against spurious correlations in an interpretable manner.