When and Why Does Bias Mitigation Work? (2023)

Abstract: Neural models have been shown to exploit shallow surface features to perform language understanding tasks, rather than learning the deeper language understanding and reasoning skills that practitioners desire. Previous work has developed debiasing techniques to pressure models away from spurious features or artifacts in datasets, with the goal of having models instead learn useful, task-relevant representations. However, what do models actually learn as a result of such debiasing procedures? In this work, we evaluate three model debiasing strategies, and through a set of carefully designed tests we show how debiasing can actually increase the model’s reliance on hidden biases, instead of learning robust features that help it solve a task. Further, we demonstrate how even debiasing models against all shallow features in a dataset may still not help models address NLP tasks. As a result, we suggest that debiasing existing models may not be sufficient for many language understanding tasks, and future work should consider new learning paradigms, to address complex challenges such as commonsense reasoning and inference.

## introduction large-scale language models have established stateof-the-art performance on several language understanding tasks @xcite @xcite . however, the performance of these models can degrade when presented with examples drawn from a different distribution than their training conditions. past work has shown that these approaches are prone to exploiting spurious 'shortcuts' in the training data, instead of learning the intended task @xcite @xcite . towards the primary goal of training nlp systems that can understand natural language, the propensity of neural models to rely on misleading * *the first two authors contributed equally to this work. in this work, we critically examine this assumption. we take a step back and characterize the empirical behavior of debiasing methods at large. while prior work has largely demonstrated the effectiveness of debiasing methods by showing increased "robustness" on external challenge sets, where these improvements come from, the extent to which the model is unbiased, and the exact differences in behavior induced by debiasing procedures remain largely unknown. several factors make analyzing model behavior along these axes difficult in empirical settings. it is challenging to: (1) enumerate all possible biases in a dataset, (2) characterize task difficulty, and (3) develop fine-grained insights into the mechanisms models use to perform tasks-before and after debiasing procedures. we shed light on these questions, by constructing a suite of controlled tests to characterize the behavior of debiased models. we make our suite of synthetic tasks publicly available, providing a testbed for bias mitigation techniques. through the behaviors of debiased models on our controlled tasks, we identify several undesirable behaviors induced by debiasing procedures. first, we identify that debiasing can cause models to switch from the identified and targeted dataset bias to an unidentified dataset bias, while still seeming to demonstrate increased "robustness" to the targeted bias. this suggests that it is entirely possible that current bias mitigation approaches cause models to shift from relying on measurable biases to relying on other unknown and unidentified biases in datasets ( §5.1). further, we characterize the complexity of a dataset foot_0 and demonstrate that even when all dataset biases have been addressed by a bias mitigation procedure, the model may not be able to learn the task if it is sufficiently complex ( §5.2). this suggests that debiasing approaches alone are unlikely to help address complex tasks such as those involving commonsense reasoning. our contributions are: ## experimental setup with the above definitions in place, our hypotheses are the following: (1) debiasing procedures can cause pretrained language models to simply switch from relying on a measurable and targeted bias to relying on unknown, unmeasurable biases, (2) for higher-level complex tasks, debiasing a model does not itself help models learn tasks. ## discussion we briefly discuss our findings and recommendations for debiasing methods going forward. which debiasing technique to choose? in our work, we shed light on how different debiasing techniques can be effective based on properties of the training dataset, including how frequent biases are, the size of the dataset as well as the amount of label noise in the data. we find that confidence regularization performs worse than the other debiasing methods in a variety of data regimes we evaluate, sometimes performing little better than the baseline. poe and reweighting often exhibit similar behavior. data filtering performs worst in highly-biased datasets. in addition, practitioners may want to consult several metrics while making the decision of which debiasing technique to choose, such as: (1) performance on out-of-domain challenge sets, (2) performance on long-tailed or rare phenomena, (3) number of additional parameters introduced by the debiasing procedure, (4) additional time required during training/inference, (5) efficiency of the debiasing procedure @xcite . ## conclusion nlu models have been shown to overrely on spurious features. this has lead to the development of several debiasing procedures to pressure models away from this behavior. in this work, we show that many of these procedures may simply push models to rely on a different set of shortcuts, and that the existing models may not be able to address complex nlu tasks even after all the spurious features have been addressed. we hope our work serves as a foundation to evaluate the effectiveness of debiasing procedures, sheds light on how these methods work, and provides an impetus for new classes of models for complex nlu tasks.