Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation (2021)

Abstract: Models pretrained with self-supervised objectives on large text corpora achieve state-of-the-art performance on English text summarization tasks. However, these models are typically fine-tuned on hundreds of thousands of data points, an infeasible requirement when applying summarization to new, niche domains. In this work, we introduce a novel and generalizable method, called WikiTransfer, for fine-tuning pretrained models for summarization in an unsupervised, dataset-specific manner. WikiTransfer fine-tunes pretrained models on pseudo-summaries, produced from generic Wikipedia data, which contain characteristics of the target dataset, such as the length and level of abstraction of the desired summaries. WikiTransfer models achieve state-of-the-art, zero-shot abstractive summarization performance on the CNN-DailyMail dataset and demonstrate the effectiveness of our approach on three additional diverse datasets. These models are more robust to noisy data and also achieve better or comparable few-shot performance using 10 and 100 training examples when compared to few-shot transfer from other summarization datasets. To further boost performance, we employ data augmentation via round-trip translation as well as introduce a regularization term for improved few-shot transfer. To understand the role of dataset aspects in transfer performance and the quality of the resulting output summaries, we further study the effect of the components of our unsupervised fine-tuning data and analyze few-shot performance using both automatic and human evaluation.

## introduction automatic text summarization aims to distill the most salient content of a given text in a compact form. recent advances in summarization have been driven by the availability of large-scale datasets such as the cnn-dailymail (cnndm) corpus @xcite and the new york times corpus @xcite as well as by the introduction of large pretrained models such as bart @xcite and pegasus @xcite , in some cases resulting in summaries which are even favored over the human-written reference summaries. creating data for every new domain, however, is infeasible and highly costly. thus, the ability to transfer large pretrained models to new domains with little or no in-domain data is necessary, especially as such models make their way into real-world applications. unsupervised summarization approaches include autoencoders to mirror the information compression inherent in summarization @xcite @xcite as well as large-scale pretraining for domainspecific adaptation @xcite . however, little work has focused on domain adaptation in summarization. @xcite examine domain adaptation for extractive summarization. @xcite showed that summarization models have difficulty generating text in the style of the target domain, while more recently, @xcite report strong performance of pretrained models when trained in few-shot settings and @xcite fine-tune datasetspecific components of a model for few-shot learning. we aim to build on recent work in pretrained models and improve zero-shot and few-shot summarization by encoding characteristics of the target summarization dataset in unsupervised, intermediate fine-tuning data. summarization can be seen as a function of subfunctions of the input, called subaspects, which determine the output form. @xcite define three subaspects for summarization: position, importance, and diversity, and study how these subaspects manifest themselves in summarization corpora and model outputs. for example, a common subaspect for the cnndm dataset is position; earlier sentences tend to constitute a good sum-mary. inspired by this view of summarization as subaspects, we aim to encode subaspects of a target dataset into unlabeled data to allow a model finetuned on this data to learn characteristics of the target dataset to improve zero-shot and few-shot transfer of the model. in our work, we focus on the subaspects of extractive diversity, as determined by how well an extractive model performs on the data, compression ratio between the source document and summary, and, in the case of cnndm, the lead bias. we assume knowledge of the target dataset such as the size of input documents, the size of the desired summaries, and the extent to which the summary is abstractive, all of that can be treated as prior knowledge if the task is to be well-defined @xcite . we encode this knowledge into wikipedia article data by extracting summaries of the desired output length and filtering examples based on the desired level of abstraction. our contributions are the following: 1) we introduce a novel method, called wikitransfer, which creates pseudo-summaries with subaspects of the target dataset which can be used as unlabeled data for intermediate fine-tuning. we show that this method improves zero-shot domain transfer over transfer from other domains, achieving state-of-theart unsupervised abstractive summarization performance on the cnndm dataset while generalizing to other domains, and we perform extensive hyperparameter studies on the factors influencing zeroshot performance 2) we demonstrate the benefits of wikitransfer in few-shot settings, and show additional improvements when applying wikitransfer with data augmentation and a regularization term for training with potentially noisy augmented data. we show robustness in these settings and analyze differences in performance in both automatic and human assessments. ## related work while advances have been made in neural techniques for summarization due in part to large datasets, less work has focused on domain adaptation of such methods in the zero and few-shot settings. @xcite examine domain adaptation, but in extractive summarization. @xcite examine domain adaptation between opinion and news summarization, observing that models trained on one domain and applied to another domain can capture relevant content but differ in style in generating the summary. @xcite introduce plug-in networks, small finetune-able layers that aim to reproduce characteristics of the target dataset as seen in a small set of labeled examples. in contrast, we aim to encode the characteristics of our target dataset, such as level of extraction and compression, a priori in the intermediate training phase. in other work, @xcite adapt a single-document summarization model to multi-document settings, while @xcite use wikipedia reference data for downstream query-based summarization several approaches for unsupervised summarization have made use of variational autoencoders @xcite @xcite . @xcite makes use of pretrained language models for unsupervised text summarization by aligning the coverage of the generated summary to the source document. @xcite train an unsupervised summarization model with reinforcement learning rewards. in another line of work, extractive models such as textrank, @xcite , lexrank @xcite , and more recently pac-sum @xcite , make use of graph centrality for modeling salience. the power of pretrained models for few-shot transfer was shown for abstractive summarization in @xcite and extractive summarization in @xcite . our work focuses on the zero-shot abstractive summarization setting and the transferability of models fine-tuned on taskspecific data from a generic corpus, rather than just the transferability of a single pretrained model. the closest work to ours for zero-shot transfer is yang et al. (2020), which uses the lead-bias in news to pretrain an unsupervised model on a large dataset of news articles. our approach, however, focuses on fine-tuning an already-pretrained model specifically for summarization on a downstream dataset by leveraging a generic text corpus (wikipedia) to create auxiliary fine-tuning data that transfers across domains, allowing for more fine-grained control over the transfer process. we show the generalizability of such fine-tuning across domains. bart @xcite is a pretrained denoising autoencoder and achieved state-of-the-art performance when fine-tuned on summarization tasks at the time. in this work, we use bart as our base pretrained model but in future work will experiment with other pretrained models. ## experimental settings datasets: we experiment with four datasets, cn-ndm, xsum @xcite , reddit_tifu (reddit) @xcite , and bigpatent @xcite . the datasets were chosen as they all differ in their abstractiveness, output length (from one sentence in xsum to on average four in big-patent), and cover multiple domains from news (cnndm and xsum) to social media (reddit) to patent documents (bigpatent), to show the generalizability of our results. each of the datasets falls into a different extractive bin, from the most extractive cnndm to the more abstractive xsum; we discuss these settings further in the appendix. ## zero-shot transfer results we compare the zero-shot performance of bart fine-tuned on wikitransfer data to that of one transferred from other summarization datasets. we also show the effect of different choices for wikitransfer fine-tuning data on cnndm and xsum. ## few-shot transfer results we examine whether zero-shot transfer improvements also carry over to the few-shot setting. also, we explore the effect of data augmentation and consistency regularization techniques. the results of our experiments with varying training data sizes and augmentation methods for all 4 datasets are shown in figure 1 and the appendix. transfer with consistency training: we find contrasting trends with the added consistency loss compared to data augmentation via round-trip translation. we note the most sizeable improvements in the more abstractive cases of xsum and reddit. we hypothesize that the consistency loss promotes better abstraction as the model learns to be invariant to noise which does not change the meaning of the text, and is thus equipped with a better notion of paraphrasing. the consistency loss allows for better training of vanilla bart as well as in general better transfer from other domains than without consistency loss. the loss likely provides a regularization factor which prevents the models from overfitting to the supervised examples. as the wikitransfer model is already more closely tuned to the target domain, this regularization may not make as large of a difference. this aligns with our observation of wikitransfer models being more robust to noisy backtranslated data on xsum and reddit. transfer to reddit shows similar results across models for consistency loss with 100 examples (better rouge-l for wikitransfer, better rouge-1/2 for reddit); vanilla bart's strong performance at 100 examples suggests that the information provided in this subset is sufficient for good performance, thus diminishing the gains from the head-start the wikitransfer model provides in zero and 10-shot transfer. we leave aspects of the consistency training such as the role of the quality of the round-trip translation data and its relation to the transfer domain to future work. ## conclusion we introduced wikitransfer, a novel and generalizable method for fine-tuning pretrained models on dataset-specific unsupervised data obtained from generic wikipedia data. wikitransfer models achieve state-of-the-art zero-shot abstractive summarization performance on the cnn-dailymail dataset and generalize across three additional datasets. in few-shot settings, wikitransfer models are robust to noise introduced through data augmentation and benefit from consistency loss on more abstractive datasets. furthermore, human assessments of the resulting summaries do not show significant differences between the wikitransfer fewshot summaries and fully-supervised summaries, demonstrating the efficiency of our approach.