Summarization-based Data Augmentation for Document Classification (2023)

Abstract: Despite the prevalence of pretrained language models in natural language understanding tasks, understanding lengthy text such as document is still challenging due to the data sparseness problem. Inspired by that humans develop their ability of understanding lengthy text form reading shorter text, we propose a simple yet effective summarization-based data augmentation, SUMMaug, for document classification. We first obtain easy-to-learn examples for the target document classification task by summarizing the input of the original training examples, while optionally merging the original labels to conform to the summarized input. We then use the generated pseudo examples to perform curriculum learning. Experimental results on two datasets confirmed the advantage of our method compared to existing baseline methods in terms of robustness and accuracy. We release our code and data at https://github.com/etsurin/summaug.

## introduction although the pretrained language models @xcite @xcite have boosted the accuracy of various natural language understanding tasks, the accuracy is still limited for complex tasks with lengthy input @xcite and fine-grained output @xcite , such as document classification. these tasks require models to find a mapping between diverse input and output, which models are more likely to suffer from the data sparseness problem. to address the data sparseness problem, researchers have studied data augmentation for text classification tasks. a basic approach is to generate pseudo training examples from gold examples by perturbing the inputs; those perturbation include back-and-forth translation @xcite and minor editing of input text @xcite or its hidden representations @xcite @xcite . these methods basically echo the information in the original training data, which will not help much the model learn to read lengthy inputs. in this study, to effectively develop the model's ability to comprehend the content in document classification, we propose a simple yet effective summarization-based data augmentation, sum-maug, to generate pseudo, abstractive training examples for document classification. specifically, we apply text summarization to the input of gold examples in document classification task to obtain abstractive, easy-to-read examples, and merge finegrained target labels as needed so that the labels conforms to the summarized input. motivated by that we humans gradually develop the ability of understanding lengthy text from reading shorter text, we use the generated examples in the context of curriculum learning (surveyed in @xcite ), namely, curriculum fine-tuning. we compare our method to a baseline data augmentation @xcite on two versions of imdb dataset with a different number of target labels. experimental results confirm that curriculum fine-tuning with summaug outperforms baseline methods on both accuracy and robustness. ## related work in this section, we first review existing neural models for document classification, and next introduce existing data augmentation methods for text classification. we then mention other attempts to leverage summarization for text classification. ## summaug document classification requires a model to comprehend lengthy text with dozens of sentences, which is even difficult for humans, especially, children and second-language learners. then, how do we humans develop an ability to comprehend lengthy text? in school, starting from reading short, concise text, we gradually read longer text. in this study, we develop a summarization-based data augmentation method for document classification, summaug, and use it to generate pseudo, abstractive training examples from gold examples to perform curriculum learning in document classification. ## experiments we conduct experiments on two datasets to evaluate our method, thus demonstrating that: (1) our method shows better accuracy and robustness compared with baseline methods in both general setting and low-resource settings; and (2) curriculum fine-tuning plays an important role in achieving improvements. ## results tables 3 and 4 list the results of baseline methods and our proposed method. our method outperforms baseline methods in all experimental settings. we additionally confirm on both datasets that our data augmentation is effective even when the training data size is small. how robustly does summaug work? sum-maug achieves higher classification accuracy across datasets while improving or maintaining robustness (low standard deviations), whereas the original aeda, namely aeda (mixed), reduces the accuracy on imdb-2 when 200 training examples are used, and it leads to unstable results on imdb-10 dataset. is curriculum fine-tuning effective? we use mixed fine-tuning with summaug and curriculum fine-tuning with aeda. we observe that under mixed fine-tuning method, the data augmented by summaug exhibited less improvements and even turns to be harmful on the imdb-10 dataset. conversely, it turns out that curriculum learning helps the aeda method achieve further improvements in some cases while addressing the low robustness issue. however, curriculum learning with aeda does not consistently enhance results because the aeda augmented data retains the same information as the original data, which offers limited benefits in improving text comprehension. ## conclusion and future work this study explores a novel application of a summarization model and proposes a simple yet effective data-augmentation method, summaug, for document classification. it performs curriculum learning-style fine-tuning to first train a model on concise summaries prior to the fine-tuning on the original training data. this mirrors the human process of mastering lengthy text comprehension, through gradual exposure to longer text. experimental results on two document classification datasets confirm that summaug enhances both accuracy and training stability compared to the baseline data augmentation method. meanwhile, our method shows effective in low-resource settings. the future work will focus on searching for the optimal mapping function f and exploring the effect of different summarization models. we will also apply summaug to other document classification tasks of various domains.