Learning to Model and Ignore Dataset Bias with Mixed Capacity Ensembles (2020)

Abstract: Many datasets have been shown to contain incidental correlations created by idiosyncrasies in the data collection process. For example, sentence entailment datasets can have spurious word-class correlations if nearly all contradiction sentences contain the word “not”, and image recognition datasets can have tell-tale object-background correlations if dogs are always indoors. In this paper, we propose a method that can automatically detect and ignore these kinds of dataset-specific patterns, which we call dataset biases. Our method trains a lower capacity model in an ensemble with a higher capacity model. During training, the lower capacity model learns to capture relatively shallow correlations, which we hypothesize are likely to reflect dataset bias. This frees the higher capacity model to focus on patterns that should generalize better. We ensure the models learn non-overlapping approaches by introducing a novel method to make them conditionally independent. Importantly, our approach does not require the bias to be known in advance. We evaluate performance on synthetic datasets, and four datasets built to penalize models that exploit known biases on textual entailment, visual question answering, and image recognition tasks. We show improvement in all settings, including a 10 point gain on the visual question answering dataset.

## introduction modern machine learning algorithms have been able to achieve impressive results on complex tasks such as language comprehension or image understanding. however, recent work has cautioned that this success is often partially due to exploiting incidental correlations that were introduced during dataset creation, and are not fundamental to the tar- * work completed at the university of washington † work completed at the allen institute for ai get task. examples include textual entailment models learning the word "not" always implies contradiction @xcite , visual question answering (vqa) models learning "2" is almost always the answer to "how many" questions @xcite , and question answering models selecting entities that occur near question words irrespective of context @xcite . we call these kinds of dataset-specific correlations dataset bias. models that exploit dataset bias can perform well on in-domain data, but will be brittle and perform poorly on out-of-domain or adversarial examples. prior work @xcite @xcite has shown it is possible to prevent models from adopting biased methods, but require the bias to be known and carefully modeled in advance, e.g., by assuming access to a pre-specified classifier that uses only the bias to make predictions. in this paper, we present a debiasing method that can achieve similar results, but that automatically learns the bias, removing the need for such datasetspecific knowledge. to make this possible, we observe that many known examples of dataset bias involve models learning overly simple patterns @xcite @xcite . this leads us to propose that many dataset biases will be shallower and easier to model than more generalizable patterns. this reflects the intuition that high-quality models for tasks like language comprehension or image understanding will require some minimum amount of complexity (e.g., a visual question answering model should at least consider the question, image, and ways they might correspond), and therefore shallower approaches are likely to be modelling dataset bias. our method, called mixed capacity ensembling (mce), follows prior work @xcite by training an ensemble of two mod-figure 1 : an overview of our method. we train a lower capacity model in an ensemble with a higher capacity model. during training simplistic correlations (e.g., "grass is usually green") are captured by the lower capacity model, which frees the higher capacity model to focus on more robust patterns (i.e., matching the question with the image). at test time, the higher capacity model is used alone. we use an independently optimized classifier as the final layer of each model as part of our method to make them conditionally independent (section 2.4). we evaluate our method by training models on datasets with known biases, and then testing them on out-of-domain datasets built to penalize models that learn to use those biases. first, we construct a series of synthetic datasets to show our method can adapt to multiple kinds of biases. then, we consider three datasets from prior work that test against question-type biases for visual question answering @xcite and hypothesis keyword biases @xcite or lexical overlap biases @xcite for sentence entailment. finally, we construct an image recognition dataset using imagenet @xcite ) that includes a test set of examples with misleading backgrounds (e.g., a fish photographed on dry land) to test our method on background-class biases. we show improved performance in all settings, in some cases nearly matching the results that can be achieved with an upper-bound that does use knowledge of the bias being targeted. we release our datasets and code to facilitate future work. foot_0 ## mixed capacity ensembles in this section, we present our mixed capacity ensembling method and the motivations behind it. we also discuss an extension to cases where shallow patterns can partially solve the task by eliminating obviously wrong answers. ## experimental setup we apply our method to datasets that have known biases, and evaluate the trained model on adversarial datasets that were constructed to penalize models that make use of those biases. this is not a perfect evaluation because models might be unfairly penalized for ignoring biases that were unaccounted for when the adversarial dataset was constructed, and therefore still exist in the test set. however, positive results on these cases provide good evidence that our method was at least able to identify the bias the adversarial dataset targeted. we use standard, high-performing models for the higher capacity models. to help ensure the lower capacity models can capture a wide range of possible biases, we pick models that achieve strong results if trained on their own, while still having significantly fewer (i.e., half or less) parameters then the higher capacity model. for each dataset, we evaluate on an out-ofdomain (ood) and in-domain (id) test set. all reported results are averages of ten runs unless otherwise specified. we provide overviews of the datasets and models used, but leave other training details to the appendix. ## related work prior work has shown that, given precise knowledge of a dataset bias, it is possible to train a debaised model. for example, if it is known particular intermediate representations in a network could only contain features that are shallowly indicative of the class due to bias (e.g., a local patch in an image), adversarial networks can be used @xcite @xcite . alternatively, given a model that is so tightly constrained it can only utilize dataset bias (e.g., a question only model for vqa), rebi @xcite ) employs a conditional independence penalty between that model and a debaised model, hex @xcite constructs a feature space that is orthogonal to the one learned by the bias model, and @xcite and @xcite pre-train the bias model and then train a debiased model in an ensemble with it. our approach also makes use of ensembling, and the idea of requiring conditional independence between the models. however, our method identifies biased strategies during training instead of requiring them to be pre-specified. additional work has used multi-label annotations @xcite , pixel-level annotations @xcite , or other annotated features @xcite to help train debiased models, although these kinds of annotations will not always be available. there are also debiasing strategies that identify hard examples in the dataset and re-train the model to focus on those examples @xcite @xcite . this approach reflects a similar in-tuition that simplicity is connected to dataset bias, although our method is able to explicitly model the bias and does not assume a pool of bias-free examples exist within the training data. another debiasing approach is to use pretraining @xcite or carefully designed model architectures @xcite @xcite to make models more prone to focus on semantic content. we expect these methods to be complementary to our work; for instance we show we can improve the performance of the extensively pretrained bert and lxmert models. a related task is domain generalization, where the goal is to generalize to a unseen test domain given multiple training datasets from different domains @xcite . most domain generalization methods learn a data representation that is invariant between domains through the use of domain-adversarial classifiers @xcite , ensembling domain-specific and domain-invariant representations @xcite or other means @xcite @xcite . our approach is similar to the ensembling methods in that we explicitly model generalizable and non-generalizable patterns, but does not require multiple training datasets. ## conclusion we have presented a method for improving outof-domain performance by detecting and avoiding overly simple patterns in the training data. our method trains an ensemble of a lower capacity model and a higher capacity model in a way that encourages conditional independence given the label, and then uses the higher capacity model alone at test time. experiments show this approach successfully prevents the higher capacity model from adopting known biases on several real-world datasets, including achieving a 10-point gain on an out-of-domain vqa dataset.