A Multi-Level Optimization Framework for End-to-End Text Augmentation (2022)

Abstract: Text augmentation is an effective technique in alleviating overfitting in NLP tasks. In existing methods, text augmentation and downstream tasks are mostly performed separately. As a result, the augmented texts may not be optimal to train the downstream model. To address this problem, we propose a three-level optimization framework to perform text augmentation and the downstream task end-to- end. The augmentation model is trained in a way tailored to the downstream task. Our framework consists of three learning stages. A text summarization model is trained to perform data augmentation at the first stage. Each summarization example is associated with a weight to account for its domain difference with the text classification data. At the second stage, we use the model trained at the first stage to perform text augmentation and train a text classification model on the augmented texts. At the third stage, we evaluate the text classification model trained at the second stage and update weights of summarization examples by minimizing the validation loss. These three stages are performed end-to-end. We evaluate our method on several text classification datasets where the results demonstrate the effectiveness of our method. Code is available at https://github.com/Sai-Ashish/End-to-End-Text-Augmentation .

## introduction data augmentation @xcite @xcite is an effective technique for mitigating the deficiency of training data and preventing overfitting. in natural language processing, many data augmentation methods have been proposed, such as back translation @xcite , synonym replacement @xcite , random insertion @xcite , and so on. in existing approaches, * corresponding author data augmentation and downstream tasks are performed separately: augmented texts are created first, then they are used to train a downstream model. the downstream task does not influence the augmentation process. as a result, the augmented texts may not be optimal for training the downstream model. in this paper, we aim to address this problem. we propose an end-to-end learning framework based on multi-level optimization @xcite , which performs data augmentation and downstream tasks in a unified manner where not only augmented texts influence the training of the downstream model, but also the performance of downstream task affects how data augmentation is performed. in our framework, we use a text summarization @xcite model to perform data augmentation. given an original text t with class label c, we feed t into the summarization model to generate a summary s. we set the class label of s to be c. @xcite is treated as an augmented text-label pair of (t, c). the motivation of using a summarization model for text augmentation is two-fold. first, the major semantics of an original text is preserved in its summary; therefore, it is sensible to assign the class label of the original text to its summary. second, the summary excludes non-essential details in the original text; as a result, the semantic diversity between the summary and the original text is rich, which well serves the purpose of creating diverse augmentations. the summarization model is trained on a summarization dataset {(t i , s i )} m @xmath0. for the downstream task, we assume it is text classification. we assume there is a text classification training set {(x (tr) i , y (tr) i )} n (tr) @xmath1, and there is a text classification validation set {(x (val) i , y @xcite our framework consists of three learning stages that are performed end-to-end. we train a text summarization model g on the summarization dataset at the first stage. considering a domain difference between the summarization data and text classification data, we associate each summarization training pair with a weight @xmath2, 1]. a smaller a indicates a large domain difference between this summarization pair and the text classification data, and this pair should be down-weighted during the training of the summarization model. these weights are tentatively fixed at this stage and will be updated later. at the second stage, we use the trained summarization model to perform text augmentation for the classification dataset and train a text classification model on the augmented and original datasets. at the third stage, we validate the classification model trained at the second stage and update weights of summarization training examples by minimizing the validation loss. the three stages are performed end-to-end where they mutually influence each other. we evaluate our framework on several text classification datasets. various experiments demonstrate the effectiveness of our method. the major contributions of this work include: the rest of the paper is organized as follows. section 2 reviews related work. section 3 introduces the method. section 4 gives experimental results. section 5 concludes the paper. ## method this section proposes a three-level optimization framework to perform end-to-end text augmentation. ## experiments in this section, we report experimental results. the key takeaways include: 1) our proposed end-to-end augmentation method performs better than baselines that conduct augmentation and classification separately; 2) our framework is agnostic to the choices of classification models and can be applied to improve a variety of classifiers; 3) our framework is particularly effective when the number of training examples is small; 4) our framework is more effective when the length of input texts is large. ## conclusions and discussion in this paper, we propose a three-level optimization framework to perform text augmentation and classification end-to-end. our framework consists of three learning stages performed end-to-end: 1) training a text summarization model; 2) training a text classification model; and 3) updating weights of summarization examples by minimizing the validation loss of the classification model. each learning stage corresponds to one level of the optimization problem in the framework. the three levels of optimization problems are nested and solved in a unified way. our framework enables the augmentation process to be influenced by the performance of the text classification task so that the augmented texts are specifically suitable for training the classification model. experiments on various datasets demonstrate the effectiveness of our method. our framework can be extended to other downstream tasks beyond classification, including but are not limited to text-to-image generation, visual question answering, dialog generation, etc. to extend our framework to a downstream task t , we need to change the loss functions in eq. ( 2 ) and eq. ( 3 ) to the loss of task t . for example, to apply our framework for text-to-image generation, given each text-image (t, i) pair, we perform augmentation of the input text t using the summarization model trained in eq. ( 1 ), to get an augmented text t. ( t, i) would be treated as an augmented data pair. then we define gan-based losses @xcite on each ( t, i) and each (t, i) to train a text-to-image generation model. we plan to study such extensions in future work. our current framework has the following limitations. first, it incurs additional computation and memory costs due to the usage of a summarization model. second, currently, our method uses a text summarization model to generate augmentations. publicly available summarization datasets are limited in size, limiting the augmentation model's quality. we plan to address these limitations in future works. to reduce memory and computation costs, we will perform parameter sharing where the encoder weights of the summarization model and those of the classification model are tied together. we plan to leverage data-rich text generation tasks for augmentation to address the second limitation, such as using machine translation models to perform back translation.