E xtra P hrase: Efficient Data Augmentation for Abstractive Summarization (2022)

Abstract: Neural models trained with large amount of parallel data have achieved impressive performance in abstractive summarization tasks. However, large-scale parallel corpora are expensive and challenging to construct. In this work, we introduce a low-cost and effective strategy, ExtraPhrase, to augment training data for abstractive summarization tasks. ExtraPhrase constructs pseudo training data in two steps: extractive summarization and paraphrasing. We extract major parts of an input text in the extractive summarization step and obtain its diverse expressions with the paraphrasing step. Through experiments, we show that ExtraPhrase improves the performance of abstractive summarization tasks by more than 0.50 points in ROUGE scores compared to the setting without data augmentation. ExtraPhrase also outperforms existing methods such as back-translation and self-training. We also show that ExtraPhrase is significantly effective when the amount of genuine training data is remarkably small, i.e., a low-resource setting. Moreover, ExtraPhrase is more cost-efficient than the existing approaches

## introduction neural encoder-decoders have achieved remarkable performance in various sequence-to-sequence tasks including machine translation, summarization, and grammatical error correction @xcite @xcite . recent studies indicated that neural methods are governed by the scaling law for the amount of training data @xcite . in short, the more training data we prepare, the better performance a neural model achieves. in this paper, we address increasing the training data for summarization to improve the performance of neural encoder-decoders on abstractive summarization tasks. in sequence-to-sequence tasks, we need a parallel corpus to train neural encoder-decoders. since it is too costly to construct genuine (i.e., humangenerated) parallel corpora, most studies explored the way to construct pseudo training data automatically. back-translation is a widely used approach to construct pseudo training data for sequence-tosequence tasks @xcite @xcite . in the backtranslation approach, we construct a model generating a source side sentence from a target side sentence, and apply the model to a target side corpus to generate a pseudo source side corpus. in addition to machine translation, back-translation is also used in grammatical error correction @xcite and summarization @xcite tasks. however, back-translation on summarization is an unrealistic problem because a model is required to restore deleted information in the given summary without any guide. @xcite indicated that the self-training approach, which makes a model generate target sentences from source sentences and use the pairs to train a model, can improve the performance on machine translation and summarization. however, pseudo data generation for summarization by self-training is hard to generate diverse summaries @xcite . moreover, self-training and back-translation approaches require expensive computational cost because we need to train additional neural encoder-decoders on a large amount of training data to obtain high-quality pseudo data @xcite . to solve these issues, we propose a novel strategy: extraphrase consisting of extractive summarization and paraphrase to construct pseudo training data for abstractive summarization. firstly, ex-traphrase extracts an important part from a source text as a summary without requiring additional model training. then, we apply a paraphrasing technique to the extracted text to obtain diverse chinese men's team have put a major step towards a new title at the military pentathlon world championships. chinese men's team have put a major step towards a new title. chinese men's team has taken a big step towards a new title. chinese men's team have put a step major towards a title at championships the world military pentathlon new chinese men's team have put a major step towards a new title. chinese men's team has taken a big step towards a new title. die chinesische herrenmannschaft hat einen großen schritt in richtung eines neuen titels gemacht. ## proposed method: extraphrase as described in section 1, our extraphrase consists of two steps: extractive summarization and paraphrasing. figure 1 illustrates the overview of ex-traphrase briefly. extraphrase receives a (genuine) sentence as an input, and generates a pseudo summary corresponding to the input sentence. when we construct a pseudo summary from a document, we independently apply extraphrase to multiple sentences included in the given document. ## experiments to investigate the effect of extraphrase, we conduct experiments on two summarization tasks: headline generation and document summarization tasks. ## efficiency of pseudo-data generation our proposed extraphrase does not require additional neural encoder-decoders such as the backtranslation and self-training approaches. we discuss the advantage of this property. in self-training, we train a model on genuine data and apply it to generate pseudo data. @xcite applied self-training to enlarge parallel corpus for neural machine translation. he et al. (2020) introduced noisy self-training that uses dropout as the noise while decoding in self-training. these studies reported the effectiveness of selftraining but self-training is hard to generate diverse pseudo data @xcite . perturbation using perturbation that is a small difference from genuine data can be regarded as data augmentation @xcite . @xcite investigated the performance of various perturbations including adversarial perturbations @xcite , word dropout @xcite , and word replacement on various sequence-to-sequence tasks. since these perturbations are orthogonal to our extraphrase, we can combine them with ours. in fact, @xcite reported that simple perturbations such as word dropout are useful on pseudo data generated by back-translation. ## conclusion this paper proposes a novel strategy, extraphrase, to generate pseudo data for abstractive summarization tasks. extraphrase consists of two steps: extractive summarization and paraphrasing. we obtain the important parts of an input by the extractive summarization, and then obtain diverse expressions by the paraphrasing. experimental results indicate that extraphrase is more effective than other pseudo data generation methods such as back-translation and self-training. moreover, we show that extraphrase is much more cost-efficient than others in pseudo data construction.