Fighting Bias With Bias: Promoting Model Robustness by Amplifying Dataset Biases (2023)

Abstract: NLP models often rely on superficial cues known as dataset biases to achieve impressive performance, and can fail on examples where these biases do not hold. Recent work sought to develop robust, unbiased models by filtering biased examples from training sets. In this work, we argue that such filtering can obscure the true capabilities of models to overcome biases, which might never be removed in full from the dataset. We suggest that in order to drive the development of models robust to subtle biases, dataset biases should be amplified in the training set. We introduce an evaluation framework defined by a bias-amplified training set and an anti-biased test set, both automatically extracted from existing datasets. Experiments across three notions of bias, four datasets and two models show that our framework is substantially more challenging for models than the original data splits, and even more challenging than hand-crafted challenge sets. Our evaluation framework can use any existing dataset, even those considered obsolete, to test model robustness. We hope our work will guide the development of robust models that do not rely on superficial biases and correlations. To this end, we publicly release our code and data.

## introduction nlp models often exploit repetitive patterns introduced during data collection, known as dataset biases, to achieve strong performance @xcite . 2 this trend has led to attempts of improving the evaluation of nlp models by creating test sets that are different from the training sets, e.g., from a different domain @xcite or a different distribution @xcite , and challenge sets that focus on counterexamples to known biases in the training set, which figure 1 : to guide the development of models robust to subtle biases, we propose to extract bias-amplified splits for existing benchmarks. our approach first partitions a given dataset into biased and anti-biased instances. it then constructs a biased training set and an anti-biased test set, which are used to evaluate model generalization. to address these gaps, some works used balancing techniques to create unbiased datasets, by filtering out biased examples @xcite @xcite , or injecting anti-biased examples into the training sets @xcite . in this work we argue that in order to encourage the development of robust models, we should in fact amplify biases in the training sets, while adopting the challenge set approach and making test sets anti-biased (fig. 1 ). amplifying dataset biases might seem counterintuitive at first. our work follows recent work that challenged the assumption that biases can ever be fully removed from a given dataset @xcite , arguing that models are able to pick up on very subtle phenomena even in partially balanced (or mostly unbiased) datasets @xcite . as a result, dataset balancing, while potentially improving generalization, might make it harder to develop models that are resilient to such biases; these biases "hide" in the balanced training sets, and the way models handle them is hard to evaluate and make progress on. 3 instead, we argue that academic benchmarks should include training splits that mainly consist of biased examples (see fig. 2 ). such splits will drive the development of robust models that generalize beyond biases, ideally even subtle ones. we present a simple method to implement our approach (sec. 2). given a dataset in which both training and test sets are divided into biased and anti-biased subsets, we remove the anti-biased instances from the training set and the biased ones from the test set. the new splits then form a challenging evaluation setting. we assume that biased instances constitute the majority of a dataset @xcite , and thus the resulting training sets are similar in size to the original ones (though the test sets are smaller). to discern biased and anti-biased instances, we consider three model-based approaches (sec. 3): (a) dataset cartography @xcite , which uses training dynamics to profile the difficulty of learning individual data instances. in this approach, we identify instances that are hard-tolearn as anti-biased @xcite ; (b) partial-input models @xcite , which are forced to rely on bias, regarding instances on which they fail as anti-biased; and a method we introduce for identifying (c) minority examples @xcite , which groups a dataset's instances using deep clustering @xcite and regards the minority-label instances within each cluster as anti-biased. we apply our framework to multinli @xcite and qqp @xcite , on which trained models exceed human performance. we also experiment with two datasets that are considered more challenging: adversarial nli (anli ; @xcite and wanli @xcite . we use a roberta-base @xcite model for selecting biased and anti-biased instances according to each method, and evaluate the performance of roberta and deberta @xcite large models under our proposed setting (sec. 4). while anti-biased instances are naturally challenging for models, amplifying biases in figure 2 : different approaches to data collection. in standard datasets (1), the training and test sets mostly contain a majority of biased instances. challenge sets (2) curate anti-biased test sets. balancing and filtering methods (e.g., adversarial filtering, 3) collect unbiased training and test sets. our framework (4) contains biased training sets and anti-biased test sets. the training set makes them even more challenging; using the partial-input and minority examples methods, we observe mean absolute performance reductions of 15.8% and 31.8%, respectively. using instances detected with dataset cartography leads to smaller (though still large) reductions of 10.1%. we compare bias-amplified splits to handcrafted challenge sets such as hans @xcite , and find that our automatically-generated anti-biased test sets are both of similar difficulty to such challenge sets, and capture a more diverse set of biases. our framework can further be used to augment existing challenge sets, as training on bias-amplified data increases their difficulty. next, we investigate how many anti-biased examples are required for generalization, by gradually re-inserting such instances to the training set @xcite . while models greatly benefit from observing small amounts of anti-biased instances, anti-biased test sets remain challenging, and additional performance gains require much larger quantities (sec. 5). we then show that standard debiasing methods applied to bias-amplified training sets lead to little to no gains in performance (sec. 6). our findings may change the way we evaluate the robustness of nlp models, and in particular their level of generalization beyond the biases of their training sets. our method requires no new annotation or any task-specific expertise. it allows to rejuvenate datasets previously considered as obsolete, and thus reuse the intensive efforts used in their curation. we release our new dataset splits along with code for automatically creating biasamplified splits for other datasets. ## amplifying dataset biases to advance model robustness this section motivates our approach in view of recent developments in nlp, provides a general overview of the framework we use to implement it, and discusses its applications. ## definitions of biased and anti-biased examples our approach requires a drop-in method for classifying a dataset's examples into biased and antibiased instances. we consider the following modelbased methods for doing so. we note that none of them requires any prior knowledge or task-specific expertise. all methods can be computed automatically at the reasonable cost of training and evaluating a (possibly smaller) model on the dataset. dataset cartography @xcite is a method to automatically characterize a dataset's instances according to their contribution to a model's performance, by tracking a model's training dynamics. specifically, measuring each instance's confidence-the mean of the predicted model probabilities for the gold label across training epochs-reveals a region of easy-to-learn instances with high confidence which the model consistently predicts correctly throughout training, and a region of low-confidence hard-to-learn instances, on which the model consistently fails during training. we follow previous work which considered instances that models find easy or hard to solve as more likely to be biased or anti-biased, respectively @xcite . to estimate the confidence of test instances, we make predictions with a partially trained model at the end of each epoch on the test set (as typically done on the validation set), and use the average confidence scores across epochs. 5 to choose anti-biased examples, we use the q% most hard-tolearn instances in each of the training and the test sets individually, where q is a hyperparameter. we consider all other examples as biased. partial-input baselines is a common method for identifying annotation artifacts in a dataset. the method works by examining the performance of models that are restricted to using only part of the input. such models, if successful, are bound to rely on unintended or spurious patterns in the dataset. examples include question-only models for visual question answering @xcite , endingonly models for story completion @xcite and hypothesis-only models for natural language inference @xcite . held-out instances where such baselines fail are considered anti-biased and less likely to contain artifacts @xcite . 6 generating a biased training set for this method is not trivial, as the partial-input model is likely to fit to the training data during training, and thus almost all examples will be labeled biased. we therefore follow the dataset cartography approach with a partial-input baseline, and compute the mean confidence score for each instance across epochs. we select the q% most hard-to-learn instances as anti-biased. minority examples current models are typically sensitive to minority examples that defy common statistical patterns found in the rest of the data, especially when the amount of such examples in the training set is scarce @xcite . minority examples are often detected by heuristically searching for spurious features correlated with one label in the instances of another label (e.g., high word overlap between two nonparaphrase texts). motivated by recent work that leverages instance similarity in the representation space of fine-tuned language models for various use cases @xcite , we propose a model-based clustering approach to automatically detect minority examples. we follow a three-step approach. first, we cluster the training set using [cls] token representations extracted from a model trained on the dataset. second, to detect minority examples in the training set, we inspect the distribution of instances over the task labels l within each cluster c i . we define a cluster c i 's majority label as the label ℓ i p l associated with the most instances in c i . we consider all other labels as c i 's minority labels. instances belonging to their cluster's minority labels are regarded as minority examples, and accordingly anti-biased, while and all others are considered biased. finally, to detect minority examples in the test set, we extract [cls] representations for all test instances, and assign each instance to the cluster of its nearest neighbor in the training set using euclidean distance. if the test instance px, yq is assigned to cluster c i , we consider px, yq as a majority example iff it belongs to c i 's majority label, i.e., if y "" ℓ i . 7 our preliminary experiments show that standard clustering algorithms tend to create labelhomogeneous clusters, i.e., they are less likely to cluster together instances from different labels. we thus use deepcluster @xcite , which we find to create more label-diverse clusters. deepcluster alternates between grouping a model's representations with a standard clustering algorithm foot_7 to produce pseudo-labels, and fine-tuning a new pretrained model to predict these pseudo-labels. we perform one iteration of deep clustering and then cluster the representations of the deepcluster model to obtain the final clustering. app. c shows details and preliminary results on alternative clustering methods. ## models struggle with amplified biases we next use our framework to evaluate the extent to which models generalize beyond the biases of their training sets. ## how many anti-biased examples are needed for generalization? so far, we have seen that amplifying dataset biases by eliminating all anti-biased instances from the training set uncovers shortcomings in model generalization. we next study the effect of allowing some anti-biased instances in the training set @xcite . we fine-tune roberta-large on all four datasets using the biased splits created using the minority examples method, while gradually reinserting 10%, 20%, 35%, 50% and 70% of the anti-biased instances back into the training set. 11 our results (fig. 3 ) show that reinserting 20% of the anti-biased training instances allows the model to close approximately 50% of the gap from its baseline performance on the anti-biased test set. surprisingly, performance grows slowly when restoring additional anti-biased instances, and does not match the full training set's levels even when adding 70% of anti-biased instances. this indicates that the model is capable of generalizing from small amounts of anti-biased instances, but is inefficient in gaining further improvements. results for the other models (fig. 4 ) show a similar trend. on the one hand, our results encourage careful data collection in order to fill gaps in dataset coverage @xcite . on the other hand, our findings indicate that data curation is not a sufficient solution, as models struggle on minority examples even when observing all available instances, and collecting more instances results in smaller further gains. thus, it is also necessary to develop robust models that can better generalize from biased data. our proposed framework provides a testbed for doing so. ## the effect of debiasing methods recently proposed methods were shown to be effective in improving the out-of-distribution generalization of models, either by adjusting the training loss to account for biased instances (model debiasing; @xcite , or by filtering the training set to increase the proportions of different kinds of instances found to be advantageous for generalization (data filtering; le bras et al. 2020; @xcite . we now examine whether such methods improve the generalization of models trained on bias-amplified training sets to anti-biased test instances. we consider a roberta-large model trained on bias-amplified splits of multinli and qqp based on minority examples. for model debiasing, we apply the self-debiasing framework suggested 2020 ). importantly, we apply both methods to the bias-amplified training split (rather than the original training set) and do not train on any other instances during the debiasing or filtering procedures. our results (tab. 4) show that neither debiasing nor filtering result in substantial improvements on anti-biased data. this indicates that such methods are less effective when training sets lack sufficient anti-biased instances, and highlights the need for methods that could improve model generalization when additional data curation is impractical. our findings are also in line with recent results showing that various robustness interventions struggle with improving upon standard training in real-world distribution shifts @xcite or dataset shifts @xcite . challenge sets given the exceptional performance of modern nlp tools on standard benchmarks, challenging test sets were created to better assess model capabilities across various tasks @xcite @xcite . such approaches often rely on human experts to identify model weaknesses and create challenging test cases using instance perturbations @xcite @xcite or rule-based data creation protocols @xcite . some approaches automated certain parts of these procedures, yet still require human design or annotation @xcite @xcite . inserting instances from challenge sets to the training set was shown to potentially alleviate their difficulty @xcite , perhaps similarly to how model performance in our framework improves when reintroducing anti-biased examples to the training set (sec. 5). other work extracted challenging test subsets from existing benchmarks for focused model evaluation @xcite . our framework can similarly be used to better evaluate model generalization, but without requiring additional annotations or task-specific expertise, and using data that was collected in the exact same procedure as the model's training data. we further showed (sec. 4) that our framework can be used along with existing challenge sets to increase their difficulty. ## conclusion recent approaches in nlp attempted to eliminate dataset biases from training sets to produce robust models and reliable evaluation settings, yet model generalization remains a challenge, and subtler biases persist. in this work, we argued that to promote robust modeling, models should instead be evaluated on datasets with amplified biases, such that only true generalization will result in high performance. we presented a simple framework to automatically create bias-amplified splits for a given dataset, finding that such splits are difficult for strong models when created for either obsolete or difficult datasets, and could potentially expose differences in generalization capabilities between models. our results indicate that bias amplification could ease the creation of robustness evaluation tests for new datasets, as well as inform the development of robust methods.