Rethinking Data Augmentation in Text-to-text Paradigm (2022)

Abstract: As manually labelling data can be costly, some recent studies tend to augment the training data for improving the generalization power of machine learning models, known as data augmentation (DA). With the arise of pre-trained language models (PLMs), some recent works on DA try to synthesize new samples benefiting from the knowledge learned from PLM’s pre-training. Along the same direction, we in this paper propose to integrate text-to-text language models and construct a new two-phase framework for augmentation: 1) a fine-tuning phase where PLMs are well adapted to downstream classification with the help of two novel schemes, and 2) a generation phase where the fine-tuned models are leveraged to create new samples for performance lifting. This paradigm opens up a new way of designing fine-tuning scheme to better serve DA in an easy-to-implement manner, and can be easily extended to other desired tasks. We evaluate our proposal on two public classification datasets and demonstrate its effectiveness with remarkable gains.

## introduction due to the unique challenges of natural language processing tasks, there is no one-size-fits-all da solution. most early attempts are based on token manipulation @xcite @xcite or paraphrasing @xcite , and the boost is limited or marginal, or even diminishing or adverse especially given original training corpus is relatively sufficient or the backbone classifiers are plm based, such as bert or roberta @xcite . some researchers have shifted attention on applying generative language models(glms) for da @xcite . auto-regressive generation lms such as gpt2 @xcite are capable of generating text with high fluency and diversity, and therefore could serve as generators to synthesize new samples required by classification model training. however, most existing glm-based da solutions have some drawbacks. first, they finetune glms on the training corpus of limited capacity @xcite @xcite , which can be problematic and prone to overfitting @xcite @xcite . second, how to introduce external colossal online corpus freely available, such as reviews, comments and news to benefit glms to better serve da has not been studied. third, effective fine-tuning regime customized for data characteristic and structure has rarely been studied. in addition, recent works employing fewshot in-context generation for da, such as gpt3, in avoidance fine-tuning and reap sparks of cleverness for automation, suffers from economic costs, latency in usage and lack of reliability @xcite @xcite . to meet above challenges, we explore the potential of using text-to-text (seq2seq) language models, which have proved their success in many nlp tasks such as dialogue generation and machine translations. in the context of data augmentation, the original training samples can be regarded as the source text which sheds some light on the semantic meaning of the topic, whereas new synthetic samples will be considered as the target text induced by the source. without loss of generality, we investigate the generation power of two exemplar text-to-text lms: t5 and bart. further, to cater for the text-to-text framework, we propose two fine-tuning schemes called p2p and s2s which organize the original corpus into parallel source and target text pairs, different from many studies compromised by limited labelled data, large publicly available online corpus is adopted in the fine-tuning process. the proposed solution is evaluated on two text categorization tasks. extensive experimental results prove these schemes can unleash the prowess of text-to-text generation while improving plms' generalization ability for da. ## conclusion and future work in this paper, we propose to use text-to-text lms as a new paradigm for data augmentation in text classification. compared to other methods along this direction, our approach is robust, easy-to-implement and does not need laborious human intervention. in future work, it is worth exploring more tailored fine-tuning scheme for da tasks under various scenarios.