How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers? (2020)

Abstract: Task-agnostic forms of data augmentation have proven widely effective in computer vision, even on pretrained models. In NLP similar results are reported most commonly for low data regimes, non-pretrained models, or situationally for pretrained models. In this paper we ask how effective these techniques really are when applied to pretrained transformers. Using two popular varieties of task-agnostic data augmentation (not tailored to any particular task), Easy Data Augmentation (Wei andZou, 2019) and Back-Translation (Sennrichet al., 2015), we conduct a systematic examination of their effects across 5 classification tasks, 6 datasets, and 3 variants of modern pretrained transformers, including BERT, XLNet, and RoBERTa. We observe a negative result, finding that techniques which previously reported strong improvements for non-pretrained models fail to consistently improve performance for pretrained transformers, even when training data is limited. We hope this empirical analysis helps inform practitioners where data augmentation techniques may confer improvements.

## introduction we pose the question: to what extent do modern nlp models benefit from task-agnostic data augmentations? in this paper, we provide empirical results across a variety of tasks, datasets, architectures, and popular augmentation strategies. among data augmentation techniques, we select easy data augmentation @xcite and back-translation @xcite ; eda and bt respectively. both are popular task-agnostic options, and report significant gains for lstms on a wide variety of datasets. we apply these techniques to 6 classification-oriented datasets, spanning 5 tasks with varying linguistic objectives and complexity. for fair comparison, we tune each of bert, xlnet, and roberta extensively, allocating an equal budget of trial runs to models trained with and without augmentations. as a separate dimension, we also vary the availability of training data to understand under what specific conditions data augmentation is beneficial. our findings demonstrate that these popular taskagnostic data augmentations provide only sparse and inconsistent improvements for modern pretrained transformers on many simple classification observed patterns suggest that the scale of pretraining may be the critical factor replacing the need for linguistic variety that augmentations confer. we hope our work provides guidance to ml practitioners in deciding when to use data augmentation and encourages further examination of its relationship to unsupervised pretraining. ## empirical results figure 1 shows both the baseline no da test accuracies as a reference point, and the mean relative improvement from applying eda and bt. empirically, improvements are marginal for 5 of the 6 datasets, only exceeding 1% for bert-b in a couple of instances where @xmath0. xlnet-b and roberta-b see no discernible improvements at almost any data level and just as frequently observe regressions in mean accuracy from eda or bt. mnli presents a clear outlier, with augmentations yielding relative improvements in excess of 2%, but only for bert. in contrast, the other pretrained transformers experience unpredictable, and mostly negative results. in terms of augmentation preferences for bert, bt confers superior results to eda in 60% of cases, averaging 0.18% absolute difference. this advantage is muted for both xlnet and roberta, with only 53% of cases preferring bt, and at smaller margins. finally, we examine the claim that data augmentation confers an advantage with any statistical significance. we use a one-sided t-test with null hypothesis that data augmentation confers a greater mean performance than without, using p-value of .05. examining bt and eda vs no da over all dataset and data sizes we reject the null hypothesis in 43%, 85%, and 87% of cases for bert, xl-net, and roberta respectively. moreover, for roberta, the inverse hypothesis (that no da is statistically better than da) is true in 28% of cases. we believe these results are surprising due to two advantages given to data augmentation in this ex2019 )'s results for comparison, though their setup differs slightly: they show the improvement only for eda (not the best of eda and bt), and they average over 5 classification datasets, of which we have sst-2, subj, and trec in common. ## discussion & limitations our empirical results verify that popular data augmentation techniques fail to consistently improve performance for modern pretrained transformerseven when training data is limited. a single excep-tion (bert-b on mnli) sees significant benefits from data augmentation. we speculate the outlier results could pertain to the inherent difficulty of natural language inference in low data regimes. alternatively, @xcite discuss "annotation artifacts" in mnli that lead models to rely on simple heuristics, such as the presence of the word "not" in order to make classifications. eda or bt could mitigate these spurious cues by distributing artifacts more evenly across labels. ## conclusion we examine the effect of task-agnostic data augmentation in modern pretrained transformers. isolating low data regimes (< 10k training data points) across a range of factors, we observe a negative result: popular augmentation techniques fail to consistently improve performance for modern pretrained transformers. further, we provide empirical evidence that suggests the scale of pretraining may be the primary factor in the diminished efficacy of textual augmentations. we hope our work provides guidance to ml practitioners in deciding when to use data augmentation and encourages further examination of its relationship to unsupervised pretraining. model parameters value/range fixed parameters batch size 50 optimizer adam learning rate schedule exponential decay lower case false max sequence length 100 tuned parameters num epochs [2, 20] dropout [0.05, 0.15] learning rate [1e -5, 5e -5] augmentation dose [0.5, 2.0] extra info model size (# params) 125.2m vocab size 50, 265 avg. runtime (full data) 32m