Don’t Discard All the Biased Instances: Investigating a Core Assumption in Dataset Bias Mitigation Techniques (2021)

Abstract: Existing techniques for mitigating dataset bias often leverage a biased model to identify biased instances. The role of these biased instances is then reduced during the training of the main model to enhance its robustness to out-of-distribution data. A common core assumption of these techniques is that the main model handles biased instances similarly to the biased model, in that it will resort to biases whenever available. In this paper, we show that this assumption does not hold in general. We carry out a critical investigation on two well-known datasets in the domain, MNLI and FEVER, along with two biased instance detection methods, partial-input and limited-capacity models. Our experiments show that in around a third to a half of instances, the biased model is unable to predict the main model’s behavior, highlighted by the significantly different parts of the input on which they base their decisions. Based on a manual validation, we also show that this estimate is highly in line with human interpretation. Our findings suggest that down-weighting of instances detected by bias detection methods, which is a widely-practiced procedure, is an unnecessary waste of training data. We release our code to facilitate reproducibility and future research.

## introduction several studies suggest that the impressive performance of the recent natural language understanding models might not be fully indicative of the status quo in language understanding. this is partially due to the artificial nature of the evaluation datasets which usually contain simple superficial cues, such as specific keywords or sentence structures, that spuriously correlate with the gold labels @xcite @xcite @xcite 1 https://github.com/h-amirkhani/ debiasing-assumption @xcite . such shallow patterns can be easily exploited by the model, resulting in an overestimated performance on the specific dataset and usually poor performance on other differentlyconstructed datasets. many proposals have been put forward to enhance robustness to such dataset-specific biases. some techniques rely on the sources of bias which are known a-priori for each dataset @xcite @xcite . others alleviate this requirement by identifying biased examples without explicitly modeling them, for instance by training a low-capacity model that does not go much beyond the surface patterns @xcite @xcite . the main model is then trained in a way not to rely much on the instances detected by the biased model. this is done either implicitly using the popular product of experts approach @xcite @xcite or explicitly using methods such as debiased focal loss @xcite or example reweighting @xcite . irrespective of the approach for identifying biased instances, existing bias mitigation techniques share the assumption that the main model treats the biased instances similarly to the biased model, in that it will resort to superficial biased features in case they exist. we carry out a critical investigation of the validity of this assumption. through a set of experiments we show that for a significant subset of instances, the biased model is unable to predict the main model's behavior. specifically, we find that in around a third to a half of instances in two well-known datasets in the domain, multinli @xcite and @xcite , the main model handles the input differently from two popular biased instance detection methods, partial-input and limited-capacity models. we further support this estimate through a manual validation. this highlights the need for re-thinking the discarding of instances detected by biased models, which is a widely-adopted approach in the current dataset bias mitigation techniques. ## methodology following the terminology used in the literature, we refer to the model used for detecting biased instances as biased model, the final intended model (for which we are mitigating dataset biases) as the main model, and those instances which are correctly classified by both the main and biased models as easy instances. existing bias mitigation techniques try to weaken the role of biased instances in the learning process based on the assumption that on these instances the main model behaves similarly to the biased model. we investigate this assumption by comparing the parts of input (tokens) on which the two models base their decisions. this analysis is carried out on the easy instances where the biased model is deemed to exploit superficial features. specifically, we compare the role of individual input tokens in the two models to check if the main model bases its decision on the same tokens (features) as the biased model. ## experiments with two datasets, mnli and fever, and two biased models, partial-input and tinybert, there are four experimental settings in total. we refer to the two biased models as partial and tiny. ## related work the presence of dataset-specific superficial cues have been shown in different nlp tasks including visual question answering @xcite , machine reading comprehension @xcite , natural language inference @xcite , abusive language detection @xcite , and fact verification @xcite . models that rely on these dataset biases usually have low performance in out-of-distribution settings. to measure the reliance on non-generalizable patterns, various challenging sets have been proposed @xcite @xcite @xcite . there have also been some attempts to systematically reduce dataset biases during construction @xcite . despite some bias reduction reported, the datasets may still contain hidden biased patterns @xcite ; therefore, it is crucial to empower the learning algorithms to be robust against biases. a popular approach to mitigate dataset biases is to encourage the main model to pay less attention to the instances that are correctly classified by a biased model. to train a biased model, some methods use a-priori known sources of biases. for instance, good performance given insufficient semantics of the input is attributed to bias exploitation @xcite @xcite . others try to identify biased instances without explicitly modeling them, such as by training a limited capacity model @xcite or exposing the model to only a small number of training instances (utama et al., 2020). to decrease the reliance of the model on the (likely) biased instances, some techniques implicitly reduce the updates of main model's pa-rameters for biased instances @xcite @xcite @xcite . others explicitly downweight the biased instances, for instance using debiased focal loss (mahabadi et al., 2020) or example reweighting (utama et al., 2020). ## conclusions through a set of experiments, we showed that a common core assumption of dataset bias mitigation methods does not hold for a significant portion of two widely-used benchmarks. specifically, we observed that two widely-used bias detection approaches, partial-input and low-capacity model, are unable to accurately predict model's handling of biased instances. we carried out extensive analysis and manual validation to attest the reliability of this observation. we infer that what identifies a biased instance is not the instance itself, but the way the model treats it. in other words, an instance that has evident biased patterns is not necessarily useless as long as the main model does not base its decision on these biases. it is worth noting that the dissimilarity in the handling of input between the biased and main models does not imply that the latter necessarily adopts an unbiased strategy. in other words, it is possible for the main model to treat the instance differently from the biased model but still exploits a (different) bias. as immediate future work, we plan to build on the findings of this analysis for enhancing dataset bias mitigation techniques.