SSMBA : Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness (2020)

Abstract: Models that perform well on a training domain often fail to generalize to out-of-domain (OOD) examples. Data augmentation is a common method used to prevent overfitting and improve OOD generalization. However, in natural language, it is difficult to generate new examples that stay on the underlying data manifold. We introduce SSMBA, a data augmentation method for generating synthetic training examples by using a pair of corruption and reconstruction functions to move randomly on a data manifold. We investigate the use of SSMBA in the natural language domain, leveraging the manifold assumption to reconstruct corrupted text with masked language models. In experiments on robustness benchmarks across 3 tasks and 9 datasets, SSMBA consistently outperforms existing data augmentation methods and baseline models on both in-domain and OOD data, achieving gains of 0.8% on OOD Amazon reviews, 1.8% accuracy on OOD MNLI, and 1.4 BLEU on in-domain IWSLT14 German-English.

## introduction training distributions often do not cover all of the test distributions we would like a supervised classifier or model to perform well on. often, this is caused by biased dataset collection @xcite or test distribution drift over time @xcite . therefore, a key challenge in training machine learning models in these settings is ensuring they are robust to unseen examples. since it is impossible to generalize to the entire distribution, methods often focus on the adjacent goal of out-of-domain robustness. data augmentation is a common technique used to improve out-of-domain (ood) robustness by synthetically generating new training examples 1 code is availble at https://github.com/ nng555/ssmba m x x x figure 1: ssmba moves along the data manifold m by using a corruption function to perturb an example x off the data manifold, then using a reconstruction function to project it back on. @xcite , often by perturbing existing examples in the input space @xcite . if data concentrates on a low-dimensional manifold @xcite , then these synthetic examples should lie in a manifold neighborhood of the original examples @xcite . training models to be robust to such local perturbations has been shown to be effective in improving performance and generalization in semisupervised and self-supervised settings @xcite @xcite . when the underlying data manifold exhibits easy-to-characterize properties, as in natural images, simple transformations such as translation and rotation can quickly generate local training examples. however, in domains such as natural language, it is much more difficult to find a set of invariances that preserves meaning or semantics. in this paper we propose self-supervised manifold based data augmentation (ssmba): a data augmentation method for generating synthetic examples in domains where the data manifold is difficult to heuristically characterize. motivated by the use of denoising auto-encoders as generative models @xcite , we use a corruption function to stochastically perturb examples off the data manifold, then use a reconstruction function to project them back on (figure 1 ). this ensures new examples lie within the manifold neighborhood of the original example. ssmba is applicable to any supervised task, requires no task-specific knowledge, and does not rely on class-or dataset-specific fine-tuning. we investigate the use of ssmba in the natural language domain on 3 diverse tasks spanning both classification and sequence modelling: sentiment analysis, natural language inference, and machine translation. in experiments across 9 datasets and 4 model types, we show ssmba consistently outperforms baseline models and other data augmentation methods on both in-domain and ood data. ## ssmba: self-supervised manifold algorithm 1 ssmba 1: require: perturbation function q reconstruction function r 2: input: dataset @xmath0, y 1 ) . . . (x n , y n )} number of augmented examples m 3: function ssmba(d, m) 4: train a model f on d 5: for (x i , y i ) ∈ d do 6: for @xmath1. . . m do 7: sample perturbed x ij ∼ q(x |x i ) 8: sample reconstructed xij ∼ r(x|x ij ) 9: generate ŷij ← f (x ij ) or preserve the original y i 10: end for 11: end for 12: we now describe self-supervised manifold based data augmentation. let our original dataset d consist of pairs of input and output vectors @xmath2, y 1 ) . . . (x n , y n )}. we assume the input points concentrate around an underlying lower dimensional data manifold m. let q be a corruption function from which we can draw a sample x ∼ q(x |x) such that x no longer lies on m. let r be a reconstruction function from which we can draw a sample x ∼ r(x|x ) such that x lies on m. to generate an augmented dataset, we take each pair (x i , y i ) ∈ d and sample a perturbed x i ∼ q(x |x i ). we then sample a reconstructed xij ∼ r(x|x i ). a corresponding vector ŷij can be generated by preserving y i , or, since examples in the manifold neighborhood may cross decision boundaries on more sensitive tasks, by using a teacher model trained on the original data. this operation can be repeated to generate multiple augmented examples for each input example. these new examples form a dataset that we can augment the original training set with. we can then train an augmented model on the new augmented dataset. in this paper we investigate ssmba's use on natural language tasks, using the mlm training corruption function as our corruption function q and a pre-trained bert model as our reconstruction model r. different from other data augmentation methods, ssmba does not rely on task-specific knowledge, requires no dataset-specific fine-tuning, and is applicable to any supervised natural language task. ssmba requires only a pair of functions q and r used to generate data. ## datasets to empirically evaluate our proposed algorithm, we select 9 datasets -4 sentiment analysis datasets, 2 natural language inference (nli) datasets, and 3 machine translation (mt) datasets. ## analysis and discussion in this section, we analyze the factors that influence ssmba's performance. due to its relatively small size (25k sentences), number of ood domains (3), and amount of domain shift, we focus our analysis on the baby domain within the ar-clothing dataset. ablations are performed on a single domain rather than all domains, so error bars correspond to variance in models trained with different seeds and results are not comparable with those in ## conclusion in this paper, we introduce ssmba, a method for generating synthetic data in settings where the underlying data manifold is difficult to characterize. in contrast to other data augmentation methods, ssmba is applicable to any supervised task, requires no task-specific knowledge, and does not rely on dataset-specific fine-tuning. we demonstrate ssmba's effectiveness on three nlp tasks spanning classification and sequence modeling: sentiment analysis, natural language inference, and machine translation. we achieve gains of 0.8% accuracy on ood amazon reviews, 1.8% accuracy on ood mnli, and 1.4 bleu on indomain iwslt14 @xmath3. our analysis shows that ssmba is robust to the initial dataset size, reconstruction model choice, and corruption amount, offering ood robustness improvements in most settings. future work will explore applying ssmba to the target side manifold in structured prediction tasks, as well as other natural language tasks and settings where data augmentation is difficult.