On the Impact of Data Augmentation on Downstream Performance in Natural Language Processing (2022)

Abstract: With in the broader scope of machine learning, data augmentation is a common strategy to improve generalization and robustness of machine learning models. While data augmentation has been widely used within computer vision, its use in the NLP has been been comparably rather limited. The reason for this is that within NLP, the impact of proposed data augmentation methods on performance has not been evaluated in a unified manner, and effective data augmentation methods are unclear. In this paper, we look to tackle this by evaluating the impact of 12 data augmentation methods on multiple datasets when finetuning pre-trained language models. We find minimal improvements when data sizes are constrained to a few thousand, with performance degradation when data size is increased. We also use various methods to quantify the strength of data augmentations, and find that these values, though weakly correlated with downstream performance, correlate negatively or positively depending on the task. Furthermore, we find a glaring lack of consistently performant data augmentations. This all alludes to the difficulty of data augmentations for NLP tasks and we are inclined to believe that static data augmentations are not broadly applicable given these properties.

## introduction data augmentation may be useful in situations where the data size is insufficient for the number of parameters in the model, resulting in overtraining @xcite . it has been pointed out that data augmentation does not degrade the expressive power of the model and achieves an improvement in the generalization performance of the model without adjusting the hyperparameters @xcite . while data augmentation is standard in the field of computer vision, it is not fully used in natural language processing. two factors can be cited for this. the first reason is that there has been insufficient unified validation of data augmentation methods for a wide range of datasets and data sizes. another reason is that it is still unclear what kind of data augmentation is effective for learning. in natural language processing, it is difficult to judge whether a data augmentation method is good or bad without relying on experiments, and it is necessary to search for effective data augmentations by trial and error @xcite . if it is possible to predict whether a data augmentation is effective for learning before training, it would be possible to search for data augmentations more efficiently. this paper examines the performance impact of data augmentation methods that have been proposed for natural language processing on various datasets. through this experiment, we will verify whether the data augmentation method can contribute to the improvement of performance on multiple datasets and problem settings. we also use various measures of the strength of a given data augmentation, and investigate its relationship with performance after learning. we find that although data augmentation strength (i.e. how significantly it perturbs the input) is correlated with the change in downstream performance to a given degree, its sign and degree often varies significantly. based on this, we believe that static data augmentations are not a wise choice for nlp tasks with a reasonable amount of data, and may need to be combined with data-dependent modeling innovations to be broadly applicable to future work. ## evaluation metrics and training strategies in this section, we briefly go over metrics we use to evaluate the strength of our data augmentations of a given task as well as strategies for training using data augmentations. ## results performance changes due to data augmentation ## discussion when all the original training data was used for training in the three datasets tested in this study, the effect of data augmentation on performance improvement was small, and the performance on the test data deteriorated in many cases. there are two possible reasons for this. the first is that the augmented data may have become noise. it is almost inevitable that data augmentation will result in the augmentation of sentences whose labels cannot be preserved. if some of the augmented sentences are incorrectly labeled, the quality of the dataset will deteriorate to some extent. therefore, in a setting where a relatively large number of data can be prepared, such as using all the training data, the negative impact of the decrease in data quality is stronger than the positive impact of the increase in the number of data. the second reason is that the knowledge that can be obtained by data augmentation may have already been acquired through prior learning. this is also pointed out by @xcite . therefore, for data in addition, in 1-step learning, the weaker the data augmentation, the better the performance. however, in 2-step learning, the relationship between the strength of consistent data augmentation and performance depended on the type of data set. this suggests that in 2-step learning, the effective strength of data augmentation may differ depending on the characteristics of the data set. for example, in mrpc, the difference between the data augmentation intensity and the f1 score of the baseline was negatively correlated because even trivial changes are likely to produce data that become noise in learning. in sick and sst-2, even if some of the content changes, the labels of the sentences are retained as long as the words indicating relevance and emotion remain the same. in this case, the various sentences created by strong data reinforcement in two-stage learning contribute to the learning pro-cess, allowing clean data to be learned in the second half. this may be why the difference between the strength of the data reinforcement and the f1 score from the baseline may have been positively correlated in some cases. therefore, by comparing the augmentation intensity determined by the proposed index, it may be possible to efficiently search for promising data augmentation methods before actual training. however, more work needs to be done to effectively use these methods in a practical setting. ## conclusion in this paper, we observed that most of the data augmentation methods did not improve performance when training on datasets with thousands of examples, but some of them improved performance when training on datasets with hundreds of examples. this suggests that, depending on the task and the data size, data augmentation may be effective even when a pre-trained language model is used for training. we also defined data augmentation intensity, a measure to evaluate whether data augmentation produces sentences that are different from the original sentences, and evaluated the relationship between this measure and the performance after training. as a result, the data augmentation intensity showed different correlations with the change in performance after training depending on the target dataset. for tasks with enough data, this indi-cates the limited applicability and predictability of static data augmentations. in future work, we believe the nlp community should look at modeling or adaptive learning methods @xcite to account for these differences in data.