An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models (2020)

Abstract: Recent work has shown that pre-trained language models such as BERT improve robustness to spurious correlations in the dataset. Intrigued by these results, we find that the key to their success is generalization from a small amount of counterexamples where the spurious correlations do not hold. When such minority examples are scarce, pre-trained models perform as poorly as models trained from scratch. In the case of extreme minority, we propose to use multi-task learning (MTL) to improve generalization. Our experiments on natural language inference and paraphrase identification show that MTL with the right auxiliary tasks significantly improves performance on challenging examples without hurting the in-distribution performance. Further, we show that the gain from MTL mainly comes from improved generalization from the minority examples. Our results highlight the importance of data diversity for overcoming spurious correlations.1

## introduction a key challenge in building robust nlp models is the gap between limited linguistic variations in the training data and the diversity in real-world languages. thus models trained on a specific dataset are likely to rely on spurious correlations: prediction rules that work for the majority examples but do not hold in general. for example, in natural language inference (nli) tasks, previous work has found that models learned on notable benchmarks achieve high accuracy by associating high word overlap between the premise and the hypothesis with entailment @xcite . consequently, these models perform poorly on the so-called challenging or adversarial datasets, where such correlations no longer hold @xcite @xcite . this issue has also been referred to as annotation artifacts @xcite , dataset bias @xcite , and group shift @xcite in the literature. most current methods rely on prior knowledge of spurious correlations in the dataset and tend to suffer from a trade-off between in-distribution accuracy on the independent and identically distributed (i.i.d.) test set and robust accuracy foot_0 on the challenging dataset. nevertheless, recent empirical results have suggested that self-supervised pre-training improves robust accuracy, while not using any task-specific knowledge nor incurring in-distribution accuracy drop @xcite . in this paper, we aim to investigate how and when pre-trained language models such as bert improve performance on challenging datasets. our key finding is that pre-trained models are more robust to spurious correlations because they can generalize from a minority of training examples that counter the spurious pattern, e.g., non-entailment examples with high premise-hypothesis word overlap. specifically, removing these counterexamples from the training set significantly hurts their performance on the challenging datasets. in addition, larger model size, more pre-training data, and longer fine-tuning further improve robust accuracy. nevertheless, pre-trained models still suffer from spurious correlations when there are too few counterexamples. in the case of extreme minority, we empirically show that multi-task learning (mtl) improves robust accuracy by improving generalization from the minority examples, even though preivous work has suggested that mtl has limited advantage in i.i.d. settings @xcite . this work sheds light on the effectiveness of pre-training on robustness to spurious correlations. our results highlight the importance of data diversity (even if the variations are imbalanced). the improvement from mtl also suggests that traditional techniques that improve generalization in the i.i.d. setting can also improve out-ofdistribution generalization through the minority examples. ## challenging datasets in a typical supervised learning setting, we test the model on held-out examples drawn from the same distribution as the training data, i.e., the indistribution or i.i.d. test set. to evaluate if the model latches onto known spurious correlations, challenging examples are drawn from a different distribution where such correlations do not hold. in practice, these examples are usually adapted from the in-distribution examples to counter known spurious correlations on notable benchmarks. poor performance on the challenging dataset is considered an indicator of a problematic model that relies on spurious correlations between inputs and labels. our goal is to develop robust models that have good performance on both the i.i.d. test set and the challenging test set. ## pre-training improve robust accuracy recent results have shown that pre-trained models appear to improve performance on challenging examples over models trained from scratch @xcite @xcite . in this section, we confirm this observation by thorough experiments on different pre-trained models and motivate our inquiries. models. we compare pre-trained models of different sizes and using different amounts of pretraining data. specifically, we use the bert base (110m parameters) and bert large (340m parameters) models implemented in gluonnlp @xcite pre-trained on 16gb of text @xcite . 4 to investigate the effect of size of the pre-training data, we also experiment with the roberta base and roberta large models @xcite ), foot_3 which have the same architecture as bert but were trained on ten times as much text (about 160gb). to ablate the effect of pre-training, we also include a bert base model with random initialization, bert scratch . fine-tuning. we fine-tuned all models for 20 epochs and selected the best model based on the in-distribution dev set. we used the adam optimizer with a learning rate of 2e-5, l2 weight decay of 0.01, and batch sizes of 32 and 16 for base and large models, respectively. weights of bert scratch and the last layer (classifier) of pre-trained models are initialized from a normal distribution with zero mean and 0.02 variance. all experiments are run with 5 random seeds and the average values are reported. ## improve generalization through multi-task learning our results on minority examples show that increasing the number of counterexamples to spurious correlations helps to improve model robustness. then, an obvious solution is data augmentation; in fact, both @xcite and @xcite show that adding a small number of challenging examples to the training set significantly improves performance on hans and paws. however, these methods often require task-specific knowledge on spurious correlations and heavy use of rules to generate the counterexamples. instead of adding examples with specific patterns, we investigate the effect of aggregating generic data from various sources through mtl. it has been shown that mtl reduces the sample complexity of individual tasks compared to single-task learning @xcite @xcite , thus it may further improve the generalization capability of pre-trained models, especially on the minority groups. ## discussion and conclusion our study is motivated by recent observations on the robustness of large-scale pre-trained transformers. specifically, we focus on robust accuracy on challenging datasets, which are designed to expose spurious correlations learned by the model. our analysis reveals that pre-training improves robustness by better generalizing from a minority of examples that counter dominant spurious patterns in the training set. in addition, we show that more pre-training data, larger model size, and additional auxiliary data through mtl further improve robustness, especially when the amount of minority examples is scarce. our work suggests that it is possible to go beyond the robustness-accuracy trade-off with more data. however, the amount of improvement is still limited by the coverage of the training data because current models do not extrapolate to unseen patterns. thus, an important future direction is to increase data diversity through new crowdsourcing protocols or efficient human-inthe-loop augmentation. although our work provides new perspectives on pre-training and robustness, it only scratches the surface of the effectiveness of pre-trained models and leaves many questions open, for example: why pre-trained models do not overfit to the minority examples; how different initialization (from different pre-trained models) influences optimization and generalization. understanding these questions are key to designing better pretraining methods for robust models. finally, the difference between results on hans and paws calls for more careful thinking on the formulation and evaluation of out-ofdistribution generalization. semi-manually constructed challenging data often cover only a specific type of distribution shift, thus the results may not generalize to other types. a more comprehensive evaluation will drive the development of principled methods for out-of-distribution generalization.