Boosting Text Augmentation via Hybrid Instance Filtering Framework (2023)

Abstract: Text augmentation is an effective technique for addressing the problem of insufficient data in natural language processing. However, existing text augmentation methods tend to focus on few-shot scenarios and usually perform poorly on large public datasets. Our research indicates that existing augmentation methods often generate instances with shifted feature spaces, which leads to a drop in performance on the augmented data (for example, EDA generally loses approximately 2% in aspect-based sentiment classification). To address this problem, we propose a hybrid instance-filtering framework (BoostAug) based on pre-trained language models that can maintain a similar feature space with natural datasets. BoostAug is transferable to existing text augmentation methods (such as synonym substitution and back translation) and significantly improves the augmentation performance by 2-3% in classification accuracy. Our experimental results on three classification tasks and nine public datasets show that BoostAug addresses the performance drop problem and outperforms state-of-the-art text augmentation methods. Additionally, we release the code to help improve existing augmentation methods on large datasets.

## introduction recent pre-trained language models (plms) @xcite @xcite have been able to learn from large amounts of text data. however, this also leads to a critical problem of data insufficiency in many lowresource fine-tuning scenarios @xcite @xcite @xcite . despite this, existing augmentation studies still encounter failures on large public datasets. while some studies @xcite @xcite have attempted figure 1 : the visualization of feature space shift of the laptop14 dataset based on t-sne. we calculate the shift metric s of feature space between augmented and natural instances. the augmentation methods are boostaug, monoaug, and eda augmentation, respectively. our boostaug has the least feature space shift. to leverage the language modeling capabilities of plms in text augmentation, these methods still suffer from performance drops on large datasets. to explore the root cause of this failure mode, we conducted experiments to explain the difference between "good" and "bad" augmentation instances. our study found that existing augmentation methods @xcite @xcite @xcite usually fail to maintain the feature space in augmentation instances, which leads to bad instances. this shift in feature space occurs in both edit-based and plm-based augmentation methods. for example, edit-based methods can introduce breaking changes that corrupt the meaning of the text, while plm-based methods can introduce outof-vocabulary words. in particular, for the edit-based methods, the shifted feature space mainly comes from breaking text transformations, such as changing important words (e.g., 'but ' ) in sentiment analysis. as for plm-based methods, they usually introduce out-of-vocabulary words due to word substitution and insertion, which leads to an adverse meaning in sentiment analysis tasks. to address the performance drop in existing augmentation methods caused by shifted feature space, we propose a hybrid instance-filtering framework (bo o s tau g) based on plms to guide augmentation instance generation. unlike other existing methods @xcite , we use plms as a powerful instance filter to maintain the feature space, rather than as an augmentor. this is based on our finding that plms fine-tuned on natural datasets are familiar with the identical feature space distribution. the proposed framework consists of four instance filtering strategies: perplexity filtering, confidence ranking, predicted label constraint, and a cross-boosting strategy. these strategies are discussed in more detail in section section 2.3. compared to prominent studies, bo o s tau g is a pure instance-filtering framework that can improve the performance of existing text augmentation methods by maintaining the feature space. with the mitigation of feature space shift, bo o s tau g can generate more valid augmentation instances and improve existing augmentation methods' performance, which more augmentation instances generally trigger performance sacrifice in other studies @xcite @xcite ). according to our experimental results on three finegrained and coarse-grained text classification tasks, bo o s tau g 1 significantly alleviates feature space shifts for existing augmentation methods. our main contributions are: • we propose the feature space shift to explain the performance drop in existing text augmentation methods, which is ubiquitous in full dataset augmentation scenarios. d i aug := ∅, d i boost := f i ; 5 randomly pick up k -2 folds except f i to constitute d i train ; 6 d i valid := f \ (f i d i train ); 7 use the deberta on d i train and d i valid to build the surrogate language model; 8 forall @xmath0:= f (d i org , ñ , θ); 10 forall @xmath1, c(daug), and the laug of daug; 12 if p(daug) ≥ α ∥ c(daug) ≤ β ∥ ldaug ̸ = ldorg then 13 d i aug := d i aug \ {daug}; 14 daug := daug d i aug ; 15 daug := daug d i boost ; 16 return daug ## proposed method the workflow of bo o s tau g is shown in figure 2 and the pseudo code is given in algorithm 1. different from most existing studies, which focus on unsupervised instance generation, bo o s tau g serves as an instance filter to improve existing augmentation methods. the framework consists of two main phases: 1) phase #1: the training of surrogate language models; 2) phase #2: surrogate language models guided augmentation instance filtering. the following paragraphs will provide a detailed explanation of each step of the implementation. ## related works as pretraining has advanced, text augmentation techniques have become an increasingly popular area of research @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . many of these techniques focus on lowresource scenarios @xcite @xcite @xcite @xcite . however, they tend to fail when applied to large public datasets @xcite . recent prominent works @xcite @xcite @xcite @xcite recognize the significance of pre-trained language models (plms) for text augmentation and propose plm-based methods to improve text augmentation. however, the quality of augmentation instances generated by unsupervised plms cannot be guaranteed. some re-search @xcite has attempted to use adversarial training in text augmentation, which can improve robustness, but these methods are more suitable for low-sample augmentation scenarios and cause shifted feature spaces in large datasets. while recent studies have emphasized the importance of quality control for augmentation instances @xcite @xcite , there remains a need for a transferable augmentation instance-filtering framework that can serve as an external quality controller to improve existing text augmentation methods. our work aims to address the failure mode of large dataset augmentation and improve existing augmentation methods more widely. specifically, bo o s tau g is a simple but effective framework that can work with a variety of existing augmentation backends, including eda @xcite and plm-based augmentation @xcite . ## conclusion existing text augmentation methods usually lead to performance degeneration in large datasets due to numerous low-quality augmentation instances, while the reason for performance degeneration has not been well explained. we find low-quality augmentation instances usually have shifted feature space compare to natural instances. therefore, we propose a universal augmentation instance filter framework bo o s tau g to widely enhance existing text augmentation methods. bo o s tau g is an external and flexible framework, all the existing text augmentation methods can be seamless improved. experimental results on three tc datasets and five absc datasets show that bo o s tau g is able to alleviate feature space shift in augmentation instances and significantly improve existing augmentation methods. the grammar and syntax to a certain extent. we apply the perplexity filtering strategy, but it is an implicit constraint and cannot ensure the syntax quality of the augmentation instances due to some breaking transformations, such as keyword deletions and modifications. however, we do not need precise grammar and syntax information in most classification tasks, especially in plm-based classification. for some syntax-sensitive tasks, e.g., syntax parsing and the syntax-based absc @xcite @xcite , ensuring the syntax quality of the augmented instances is an urgent problem. therefore, bo o s tau g may not be an best choice for some tasks or models requiring syntax as an essential modeling objective @xcite . in other words, the syntax quality of bo o s tau g depends on the backend. chen zhang, qiuchi li, and dawei song. 2019. aspect-based sentiment classification with aspectspecific graph convolutional networks. in emnlp-ijcnlp'19: proc. of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing, pages 4567-4577. association for computational linguistics. xiang zhang, junbo jake zhao, and yann lecun. 2015. character-level convolutional networks for text classification. in neurlips'15: advances in neural information processing systems 28: annual conference on neural information processing systems, pages 649-657. jing zhou, yanan zheng, jie tang, li jian, and zhilin yang. 2022a. flipda: effective and robust data augmentation for few-shot learning. in proc. of the 60th annual meeting of the association for computational linguistics, pages 8646-8665. association for computational linguistics. ran zhou, xin li, ruidan he, lidong bing, erik cambria, luo si, and chunyan miao. 2022b. melm: data augmentation with masked entity language modeling for low-resource ner. in acl'22: proc. of the 60th annual meeting of the association for computational linguistics, pages 2251-2262. association for computational linguistics.