Text A uto A ugment: Learning Compositional Augmentation Policy for Text Classification (2021)

Abstract: Data augmentation aims to enrich training samples for alleviating the overfitting issue in low-resource or class-imbalanced situations. Traditional methods first devise task-specific operations such as Synonym Substitute, then preset the corresponding parameters such as the substitution rate artificially, which require a lot of prior knowledge and are prone to fall into the sub-optimum. Besides, the number of editing operations is limited in the previous methods, which decreases the diversity of the augmented data and thus restricts the performance gain. To overcome the above limitations, we propose a framework named Text AutoAugment (TAA) to establish a compositional and learnable paradigm for data augmentation. We regard a combination of various operations as an augmentation policy and utilize an efficient Bayesian Optimization algorithm to automatically search for the best policy, which substantially improves the generalization capability of models. Experiments on six benchmark datasets show that TAA boosts classification accuracy in low-resource and class-imbalanced regimes by an average of 8.8% and 9.7%, respectively, outperforming strong baselines.

## introduction model performance on natural language processing (nlp) tasks, such as text classification, often heavily depends on the size and the quality of the training data. however, it is time-consuming and labor-intensive to obtain sufficient training instances, and models face low-resource regimes most of the time. data augmentation @xcite @xcite aims to enlarge the training dataset by synthesizing additional distinct and label-invariant instances based on raw instances to improve the model per- previous data augmentation methods can be roughly divided into two categories: generationbased @xcite @xcite @xcite and editing-based methods @xcite . generation-based methods utilize conditional generation models @xcite to synthesize the paraphrases @xcite of the original sentences, which have advantages in instance fluency and label preservation but suffer from the heavy cost of model pre-training and decoding. editing-based methods instead apply label-invariant sentence editing operations (swap, delete, etc.) on the raw instance, which are simpler and more efficient in practice. however, the editing-based methods are discovered plays like a volatile and overlong w magazine fashion spread. plays like a volatile and overlong w magazine fashion spread. like plays volatile and a w overlong magazine fashion spread. plays like a volatile and overlong w magazine fashion spread. given a text, we randomly pick n * operations from p then apply the operations sequentially. in this case, @xmath0. red operation o indicates it is finally applied to the text according to probability p. the modification to the original text cased by it is also showed in red. we use rs (random swap), rd (random delete), ti (tf-idf insert), ts (tf-idf substitute) for abbreviation. best viewed in color. to overcome the limitations above, we propose a framework named text autoaugment (taa) to establish a learnable and compositional paradigm for the data augmentation. our goal is to automatically learn the optimal editing-based data augmentation policy for obtaining a higher quality augmented dataset and thus enhancing the target text classification model. we design an augmentation policy as a set of various editing operations: @xmath1, • • • , op n } and each operation is defined with a parameter triplet: @xmath2, probability p, magnitude λ . such a compositional structure allows more than one operation to be applied to the original sentence and can improve the the diversity of the synthetic instances. in conclusion, a policy solution consists of two kinds of knowledge to learn: the operation set and the editing parameters for each operation. to search the optimal policy, we propose a novel objective function and utilize the sequential model-based global optimization (smbo) @xcite , an efficient and widely used method in au-toml @xcite , to learn the optimal configuration of the compositional policy. given a target dataset, our algorithm learns an augmentation policy automatically and adaptively. the implementation based on distributed learning frameworks can efficiently obtain promising results in several gpu hours. to summarize, our contribution is two-fold: ## text autoaugment in this section, we introduce the text autoaugment framework to search for the optimal augmentation policy automatically. our augmentation policy is composed of various operations and forms a hierarchical structure, which is first detailed in section 2.1. then we give an overview of the proposed learnable augmentation method and establish a global objective function for it (section 2.2). the specific policy optimization algorithm is presented at last (section 2.3). ## analysis we analyse the learned policies on four aspects including transferability, scalability, magnification and structure. besides, we evaluate the diversity and semantics preservation of the augmented data, and further conduct a case study. the final policies searched by our taa algorithm are listed in appendix d. ## related work previous data augmentation algorithms in nlp can be categorized into generation-based and editingbased methods. for generation-based methods, back-translation @xcite @xcite generates the paraphrase of a text by translating it to an intermediate language and back. @xcite masks some words then uses a label-conditional language model to predict them. anaby-tavor et al. (2020) and @xcite leverage various language models such as gpt-2 @xcite and bart @xcite ## conclusion in this paper, we propose an effective method called text autoaugment (taa) to establish a compositional and learnable paradigm for data augmentation. taa regards a combination of various editing operations as an augmentation policy and utilizes smbo for policy learning. experiments show that taa can substantially improve the generalization ability of models as well as lighten the burden of artificial augmentation designing.