A uto A ugment Is What You Need: Enhancing Rule-based Augmentation Methods in Low-resource Regimes (2024)

Abstract: Text data augmentation is a complex problem due to the discrete nature of sentences. Although rule-based augmentation methods are widely adopted in real-world applications because of their simplicity, they suffer from potential semantic damage. Previous researchers have suggested easy data augmentation with soft labels (softEDA), employing label smoothing to mitigate this problem. However, finding the best factor for each model and dataset is challenging; therefore, using softEDA in real-world applications is still difficult. In this paper, we propose adapting AutoAugment to solve this problem. The experimental results suggest that the proposed method can boost existing augmentation methods and that rule-based methods can enhance cutting-edge pretrained language models. We offer the source code.

## introduction data augmentation is a regularization strategy that improves model performance expanding the data held in various ways @xcite . in the natural language processing (nlp) field, data augmentation is used in various fields to alleviate data shortages, and various augmentation methods have been proposed accordingly @xcite . for example, image data can be augmented by applying simple rules, such as flipping and rotation, to image data @xcite , and text data can also be augmented, by simple rules such as replacing synonyms and changing the order between words @xcite . in addition, a method for augmenting data by generating new text using various deep learning models has also been proposed @xcite @xcite @xcite . however, as these methods often demand training data for fine-tuning before augmentation @xcite , it may be challenging to apply them in a low-resource environment @xcite @xcite . rule-based text data augmentation methods are less costly and easy to implement; thus, they are often used in real-world problems. despite that, the previously proposed rule-based text data augmentation methods risk not maintaining semantic consistency with original data, which is different from image data @xcite , leading to performance degradation. to relieve this problem, methods that perform data augmentation only through random insertion of punctuation marks have also been proposed @xcite , but they introduce fewer variations compared to easy data augmentation (eda). recently, softeda @xcite , a method applying label smoothing @xcite to the augmented data, was proposed to alleviate these drawbacks. in softeda, a heuristic grid search was performed for the label smoothing factor (a hyperparameter for performing label smoothing). however, the method based on a heuristic search has the following disadvantages. first, a heuristic search is expensive to execute @xcite . second, although we found the best factor value of the grid, it may not be the global optimum. there could be a better value outside the heuristic search grid; thus, revealing the possible performance gain is difficult. this paper proposes a method to apply autoaugment @xcite , a technique to determine the optimal factors in the data augmentation process to alleviate the limitations of previous soft-eda methods. by optimizing various arguments of softeda, it is shown that stable and effective performance improvement is possible compared to the existing rule-based strategy with static factors. in addition, the existing softeda experiment was conducted on an entire dataset. however, more severe overfitting occurs when the given training data are insufficient @xcite , and the scope of performance improvement is greater when additional training data are obtained from a small dataset @xcite , so data augmentation becomes increasingly crucial in this low-resource environment. therefore, this study evaluates the proposed method under a lowresource scenario and demonstrates that the proposed method is effective even under data-scarce conditions. in addition, some existing studies have argued that simple rule-based augmentation strategies are less effective in improving the performance of pre-trained language models (plms) @xcite @xcite . in this study, we show that through argument optimization, it is possible to improve the performance of not only bert @xcite , the standard plm, but also debertav3 @xcite , a cutting-edge plm, through rulebased data augmentation. ## related work data augmentation of text is primarily performed by augmenting data according to predetermined rules @xcite @xcite @xcite or using various deep learning models @xcite @xcite @xcite . rule-based data augmentation methods generate new data by performing perturbation in various ways, such as replacing some of the words in a given sentence with synonyms @xcite or inserting typos at the character level @xcite . the easy data augmentation (eda) @xcite technique is a representative rule-based data augmentation method consisting of synonym replacement (sr), random insertion (ri), random swap (rs), and random deletion (rd). however, because such random changes can lead to the loss of semantic consistency, the "an easier data augmentation" (aeda) technique @xcite consisting only of the ri of six punctuation marks has also been proposed. the softeda @xcite ) method compensates for the semantic damage caused by eda by applying label smoothing to the augmented data. model-based augmentation methods employ deep learning models to generate new data. backtranslation @xcite is one of the early model-based methods. it first translates the given data into another language and back-translates it to the original language, generating different expressions with the same concept. methods based on plm have also been proposed, and c-bert @xcite , lambada (anaby-tavor et al., 2020), and flipda @xcite generate new data using bert @xcite , gpt-2 (radford et al., 2019), and t5 @xcite , respectively. in addition, gpt-3 @xcite and chatgpt, which are larger than these plms, have been proposed to generate new data @xcite . other researchers have introduced mixup @xcite strategy to the nlp field to augment text data @xcite @xcite . moreover, some previous approaches have tried to apply autoaugment for nlp. text autoaugment @xcite , the work closest to the proposed method, suggested applying autoaugment to optimize hyperparameters for data augmentation. in addition, dnd @xcite incorporated various data augmentation methods and suggested optimizing two reward terms regarding the difficulty and consistency with the original data. while the proposed work uses autoaugment to optimize augmentation hyperparameters, we also focus on optimizing label smoothing values for the original and augmented data. ## conclusion this paper proposed a method to optimize various hyperparameters of rule-based text augmentation methods. the experimental results suggest that the proposed method is effective and stable, and that rule-based augmentation methods can improve cutting-edge plms with proper hyperparameter optimization. future work may extend this approach to other tasks, such as natural language inference, which is more complex than the single-sentence classification conducted in this paper.