Generalized but not Robust? Comparing the Effects of Data Modification Methods on Out-of-Domain Generalization and Adversarial Robustness (2022)

Abstract: Data modification, either via additional training datasets, data augmentation, debiasing, and dataset filtering, has been proposed as an effective solution for generalizing to out-of-domain (OOD) inputs, in both natural language processing and computer vision literature. However, the effect of data modification on adversarial robustness remains unclear. In this work, we conduct a comprehensive study of common data modification strategies and evaluate not only their in-domain and OOD performance, but also their adversarial robustness (AR).We also present results on a two-dimensional synthetic dataset to visualize the effect of each method on the training distribution. This work serves as an empirical study towards understanding the relationship between generalizing to unseen domains and defending against adversarial perturbations. Our findings suggest that more data (either via additional datasets or data augmentation) benefits both OOD accuracy and AR.However, data filtering (previously shown to improve OOD accuracy on natural language inference) hurts OOD accuracy on other tasks such as question answering and image classification. We provide insights from our experiments to inform future work in this direction.

## introduction deep neural networks have emerged as a widely popular architectural choice for modeling tasks in multiple domains such as (but not limited to) computer vision (yuille and @xcite , natural language processing @xcite , and audio @xcite . while these models are highly capable of learning from training data, recent studies show that they are quite prone to failure on new test sets or under distribution shift @xcite , natural corruptions @xcite , adversarial attacks (goodfellow et al., * equal contribution 2015), spurious correlations @xcite , and many other types of "unseen" changes that may be encountered after training. this shortcoming stems from the i.i.d. assumption in statistical machine learning which guarantees good performance only on test samples that are drawn from an underlying distribution that is identical to the training dataset. for instance, digit recognition models trained on the black-and-white mnist training images are almost perfect (> 99% accuracy) on the corresponding test set, yet their performance on colored digits and real-world digits from street number plates is less than 75%. similarly, state-ofthe-art nlp models have been shown to fail when negation is introduced in the input @xcite . these findings pose a significant challenge to the practical adoption of these models and their reliability in the real-world. to test model performance beyond the traditional notion of in-domain (id) generalization, two prominent ideas have emerged: out-of-domain (ood generalization) a.k.a. domain generalization foot_0 , and adversarial robustness. the ood generalization objective expects a model which is trained on distribution d to perform reliably on unseen distributions d e , @xmath0, . . . , n}, that differ from d. for a trained classifier f * , ood accuracy on previously unseen distribution d e is defined as: to define adversarial robustness, consider an input x and a true label y. for a classifier loss function ℓ, a loss-maximizing perturbation δ * within ∆ (an -bounded neighborhood of x) is defined as: the second idea is that of adversarial robustness. recent work on adversarial examples has revealed the vulnerability of deep neural networks against small perturbations of the original data. adversarial robustness in such under this setting is defined as the accuracy of the classifier on adversarial samples x +δ x , where the perturbation lies within an ℓ p norm bound: ||δ x || @xmath1. in the context of text classification, the norm-bound can also be in the form of small character-level or word-level perturbations such as swapping, inserting, or deleting characters or words. in essence, adversarial robustness measures the invariance of the classifier to small perturbations of the input. various methods have been developed that either improve ood generalization or improve adversarial robustness. notable among these are techniques that modify the distribution of the training dataset. in this paper, we focus on three major data modification techniques -the use of additional datasets (also known as multi-source training), data augmentation, and data filtering; in addition we also consider model-based debiasing techniques which do not alter the data distribution explicitly. we study the performance of these methods on three representative tasks -natural language inference (nli), extractive question answering (qa), and image classification (ic). our first aim in this paper is to understand whether the increase or decrease in ood generalization by each method over the naive baseline (standard training on the source dataset) is consistent across tasks. to further conduct fine-grained analysis, we also analyze the effect of these methods on in-domain (id) accuracy on the test set for each task, since in the ideal case improvement in ood performance should not come at the cost of in-domain accuracy. recent work seeks to understand the relationships between in-domain and out-of-domain performance: for instance, @xcite empirically show that id and ood performance are strongly correlated, @xcite ; yang et al. (2020) show a trade-off between robustness and accuracy for adversarially trained models. however it is not clear how methods designed for ood generalization affect robustness. this is largely because work on domain generalization reports only iid and ood metrics, and work on robustness reports only id and robustness metrics. our second aim is to understand the effect of these generalization methods on adversarial robustness. in addition to our experiments on nlp and vision tasks, we also provide an experiment on a synthetic binary classification dataset where points lie in a 2-dimensional feature space and are separated by concentric circles into class labels. this setting allows us to visualize the effect of data modification techniques on the training distribution and the resulting performance. our findings can be summarized as follows: these findings and our additional analysis raise new questions for robustness and domain generalization research. significant among these are the importance of both diversity and number of training samples for inductive bias and generalization guarantees, the problems associated with data filtering in terms of robustness, and the importance of a comprehensive set of evaluation metrics that could be adopted for future work. ## categorization of domain generalization methods in this section, we provide a categorization of methods that are typically used as baselines for domain generalization. we briefly explain the method and provide relevant related work in which these ideas are used as methods for domain generalization. throughout this paper, we will refer to the original training distribution as the "source" and the out-of-distribution datasets as the "targets". single-source training (ss) refers to the "vanilla" baseline which is trained only on the source dataset, without any dataset modification. ss utilizes no other information apart from the single source dataset d and updates parameters θ of classifier f to minimize the risk on the source using approaches such as erm @xcite . multi-source training (ms). this method is identical to ss except that additional training datasets d are used for risk minimization. usually d are designed for the same task as d but may have different styles, characteristics, or sources of collection. for instance, while both snli @xcite and mnli @xcite are datasets for natural language inference with identical class labels, snli was collected from image captions, while mnli was collected from open american national corpus foot_1 . gulrajani and lopez-paz (2020) provide an extensive comparitive study of models trained for multi-source domain generalization for image classification and surprisingly find that if multiple source domains are available, erm is empirically the best approach as compared to specially designed dg methods such as meta-learning @xcite , learning domain-invariant features @xcite , invariant risk minimization @xcite , etc. these findings have also been observed on text classification experiments in @xcite . @xcite show that pre-training transformer architectures on diverse data leads to higher ood accuracies on multiple tasks such as semantic textual similarity, sentiment classification, reading comprehension and natural language inference. data augmentation (da). when additional training distributions are not directly available, transformations of samples in d using pre-defined augmentation functions can be used to create d and train the model. such data augmentation functions are typically derived from existing knowledge about the invariance of the task w.r.t. certain transformations. for instance, for image classification, addition of small noise, small translations, scaling, etc. are common data augmentation functions, since they do not change the true label for the image. similarly, for text inputs, synonyms of words are commonly used since they do not change the semantics of the sentence. nlp data augmentation techniques include uda @xcite , eda @xcite , and back-translation for question answering @xcite . data filtering (df). dataset filtering has been previously explored for quality control, such as, removing noise and artifacts to curate and improve publicly sourced datasets. however, there has been recent interest in considering df as a method for bias reduction and generalization. this idea can be traced back to work by @xcite , that proposed df as an algorithmic method to avoid annotation artifacts and spurious correlations during dataset construction. aflite @xcite extended this idea to a generic filtering methodology that can work without any pre-defined rules or strategies. instead, aflite operates by utilizing several weak learners (such as support-vector machines) trained over small subsets to identify samples that are easy to classify. it is argued that such samples are more likely to carry biases, and as such, could be removed. aflite suggests that reduction of a dataset to even 10% of the original size can boost ood accuracy on nli. in the vision domain, similar ideas have been proposed concurrently, including repair @xcite and resound @xcite , in which instead of completely removing samples, biased samples are assigned smaller weights. however these methods require a prior knowledge of the bias variable. @xcite have recently proposed a simple approach which upweights samples which have higher loss -this is shown to improve worst-group accuracy without having access to the bias variable. ## toy example: concentric circles we begin with a simple two-dimensional example to illustrate our experimental setting and to show how each method affects the distribution of the training set. consider the set of points shown in figure 1 where the points belong to two class labels (either 0 or 1) and are seen to lie on concentric circles. points with label 0 are closer to the origin, while points with label 1 are closer to a distance of 1 from the origin. our aim is to start with the single source dataset and train the model to generalize on the out-of-domain (ood) dataset. an important thing to note here is that the source dataset contains a subset of points with label 0 (orange) clustered around (0.4, 0.0) and a subset with label 1 clustered around (-1, 0.0). this implies that class-0 is biased towards @xmath2, while class-1 is biased towards @xmath3. in total, our ss dataset consists of 10000 samples, of which 20% are biased. we apply three data modifications: additional source (ms), gaussian data augmentation (da) ∼n (0, 0.1), and data filtering (aflite) which reduces the dataset size to 10%. note that we do not show model debiasing (db) here, since it does not alter the data distribution. figure 2 shows the effect on the data distribution. the most striking is the effect of df which removes all samples previously in the biased clusters near (0.4, 0.0) and (-1.0, 0.0). equipped with these resulting datasets, we train a linear sgd classifier with log-loss and evaluate the robustness of each model in terms of in-domain and ood accuracies. we also evaluate adversarial robustness by using standard pgd attacks. results are shown in the textboxes in figure 2 . it can be seen that data filtering significantly hurts both ood generalization and robustness. this finding motivates our experiments to understand the effect of each method for nlp and vision tasks. ## experiments in this section, we present three tasks and their corresponding experimental setup, evaluation protocol and our findings. a summary of methods belong to each category is provided in ## analysis based on the results of three tasks, we have the following observations about the performance of each method compared to the ss baseline: decrease in nli in-domain accuracy is seen for all methods, even though these lead to increase in ood accuracy. this suggests that the training dataset (snli) has a large shift w.r.t. ood datasets. more data implies more ood generalization: while this trend is observed for both ms and da, there is one anomaly -da for the qa task leads to marginal decrease compared to ss (a difference of 0.56%). this finding is aligned with @xcite , who report no significant effect of data augmentation (back translation) on ood performance for question answering. this points to the need for improving data augmentation techniques in qa. on the other hand, the performance drop due to df is significantly large for qa (11.53%). decrease in mnist robustness: for mnist, the da method (m-ada @xcite ) is the best in terms of robustness and also improves ood accuracy. m-ada is an "adversarial data augmentation" method, i.e., it uses a min-max objective to find loss-maximizing perturbations and uses these perturbations as augmented data. it is therefore intuitive that such a method would do well on the adversarial robustness metric (although robustness evaluation was not reported by @xcite ). marginal improvement on robustness: from the results, it is easy to see that the improvement on ood is more noticeable than robustness, for example, ms improves ood performance by ∼10%, but improves only by ∼1% under model-free evaluation. while this observation is reasonable since each method is designed to improve the generalization, new methods that improve both generalization and robustness should be encouraged. ## related work in section 2 we have provided relevant work that falls into one of our five modeling categories. here, we discuss additional literature on robustness and generalization and new efforts towards dataset creation, benchmarks, and evaluation. generalization benchmarks. @xcite have constructed a robustness benchmark for multiple language understanding tasks by splitting training sets from existing benchmarks according to topics, styles, and vocabulary; this has been subsequently used to study robustness of model rankings @xcite . benchmarks have also been constructed to study dataset artifacts and generalization capabilities of models @xcite . mrqa @xcite is a benchmark for evaluating domain generalization of question answering (reading comprehensive) models. mrqa contains 6 datasets each for training, development, and evaluation. for image classification, many benchmarks have been proposed to evaluate domain generalization, such as pacs @xcite , officehome @xcite , digits @xcite which is a compendium of domain generalization bechmarks for tasks such as image classification, text sentiment and toxicity prediction. corruption robustness. @xcite introduced imagenet-c and cifar-c to test robustness along corruptions such as weather, noise, blur, and digital artifacts, and imagenet-p which tests robustness against small tilts and changes in brightness. mnist-c was introduced by @xcite for similar corruptions of handwritten digit images. adversarial and contrastive sets. generation of adversarial examples @xcite @xcite and approaches to defend against word substitution @xcite have been explored. contrastive examples have been introduced as a means for evaluation, for example, manually crafted contrast sets for textual entailment @xcite or template-based @xcite @xcite . model-in-the-loop dataset creation methods have also been proposed for various nlp tasks @xcite @xcite and visual question answering @xcite . @xcite have empirically shown linear trends between in-distribution and out-of-distribution performance on multiple image classification tasks, across various model architectures, hyper-parameters, training set size, and duration of training. they also show that there are certain settings of domain shift under which the linear trend does not hold. our work empirically shows that while data filtering may benefit ood generalization on the nli benchmark, this does not hold for other tasks such as image classification and question answering. this suggests that data filtering may benefit generalization in certain types of domain shift, but not on others. concurrently, @xcite have theoretically shown that models robust to input perturbations generalize well on ood distribution within a wasserstein radius around the training distribution. our empirical observations in this paper in both vision and language domains, agree with the theory of @xcite .