Mitigating Data Scarceness through Data Synthesis, Augmentation and Curriculum for Abstractive Summarization (2021)

Abstract: This paper explores three simple data manipulation techniques (synthesis, augmentation, curriculum) for improving abstractive summarization models without the need for any additional data. We introduce a method of data synthesis with paraphrasing, a data augmentation technique with sample mixing, and curriculum learning with two new difficulty metrics based on specificity and abstractiveness. We conduct experiments to show that these three techniques can help improve abstractive summarization across two summarization models and two different small datasets. Furthermore, we show that these techniques can improve performance when applied in isolation and when combined.

## introduction training complex neural models usually requires large amounts of data. however, data annotation still poses a challenge for many domains. thus, much research focuses on data manipulation (e.g., synthesis, augmentation) and additional ways to handle data differently during training. prior work on the synthesis of textual data has focused on back translation @xcite @xcite and word replacement @xcite . we, on the other hand, propose a different approach to data synthesis through paraphrasing. however, synthesis involves data manipulation on the input level, which might expose the model to grammatically or logically incorrect input. thus, we explore a second approach to data manipulation based on augmentation rather than synthesis. augmentation aims to move data manipulation from the input side to any part of the model. this, in turn, can help the model be more resilient to over-fitting. third, we explore using data more efficiently by integrating curriculum learning into the training process. curriculum learning reorders training samples based on external criteria, which can help train the model gradually and more efficiently without the need for any external data. we also introduce new difficulty metrics based on specificity and abstractiveness for curriculum construction. finally, we explore combining multiple techniques @xcite to overcome the data scarcity issue. thus, our contribution is threefold: 1) we introduce a simple approach for data synthesis through paraphrasing. 2) we use data augmentation by sample mixing to move augmentation into the model. 3) we integrate a curriculum into the training process and introduce two new difficulty metrics. data synthesis and augmentation. data synthesis for text summarization is underexplored, with only a few approaches such as back-generation @xcite and template-based summary re-writing @xcite . we propose doing data synthesis by paraphrasing, which is simpler than the back-translation and template methods. while combining synthesis with paraphrasing has been studied in other contexts @xcite , our work differs in both goals and techniques. @xcite proposed synthesizing data, then crowdsourcing paraphrases to train semantic parsers, while @xcite synthesized data to train a paraphrasing model. our work, to our knowledge, is the first to use a strong language model finetuned for paraphrasing to synthesize data for text summarization. finally, for data augmentation, we base our work on the mixtext approach @xcite . while the original mixtext model is used for classification-based tasks, we introduce a variation for generative tasks (called mixgen) and use it for abstractive summarization. curriculum learning. curriculum learning aims to improve the training procedure with the same amount of data. it has been applied in nlp @xcite @xcite @xcite for machine comprehension, question generation, reading comprehension, nlu and machine translation, respectively. we build on the approach introduced in @xcite ; however, the core differences are both the downstream tasks (classification versus abstractive summarization) and the difficulty metrics. in contrast to the only other summarization work that we know of, @xcite focus on large datasets, while we focus on low resource domains. we also introduce two different difficulty metrics (rouge and specificity). ## summarization datasets coursemirror (cm) is a student reflections dataset that has been used in prior work to study extractive @xcite and abstractive (magooda and litman, 2020) 1 summarization. the dataset consists of documents and summaries from four courses. amazon/yelp (a/y) is another small dataset, now consisting of opinions (refer to the appendix 1 https://petal-cs-pitt.github.io/data.html for examples) @xcite foot_0 . the dataset contains customer reviews from amazon @xcite and yelp. the data contains 160 products/businesses split into training, validation and test sets as shown in ## conclusion and future work in this work, we showed that we could mitigate the effect of data scarcity in different datasets (i.e., coursemirror and amazon/yelp) for abstractive summarization using three simple data manipulation techniques. we showed that synthesizing data with paraphrasing to use for pretraining can boost the model performance across all rouge scores for different datasets. additionally, we showed that mixing samples for training can also push the model to be more resilient to overfitting and improve its performance. finally, we showed that reordering training samples through curriculum, using the proposed difficulty metrics (i.e., specificity, and rouge) would help improve all rouge scores across different datasets without the need for any additional data (either true or synthetic). in the future, we plan to try more n values for synthesis and mixgen. additionally, we plan to investigate other curriculum difficulty metrics. we plan to use bart model as one of the sota models for abstractive summarization. finally, we are doing additional experiments on multitask learning, and we plan to combine both techniques in one framework targeting low resource domains. curriculum learning for natural language understanding. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 6095-6104, online. association for computational linguistics. tiezheng yu, zihan liu, and pascale fung. 2021. adaptsum: towards low-resource domain adaptation for abstractive summarization. in proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 5892-5904, online. association for computational linguistics. xiang zhang, junbo jake zhao, and yann lecun. 2015. character-level convolutional networks for text classification. in advances in neural information processing systems 28: annual conference on neural information processing systems 2015, december 7-12, 2015, montreal, quebec, canada, pages 649-657.