Flight of the PEGASUS ? Comparing Transformers on Few-shot and Zero-shot Multi-document Abstractive Summarization (2020)

Abstract: Recent work has shown that pre-trained Transformers obtain remarkable performance on many natural language processing tasks including automatic summarization. However, most work has focused on (relatively) data-rich single-document summarization settings. In this paper, we explore highly-abstractive multi-document summarization where the summary is explicitly conditioned on a user-given topic statement or question. We compare the summarization quality produced by three state-of-the-art transformer-based models: BART, T5, and PEGASUS. We report the performance on four challenging summarization datasets: three from the general domain and one from consumer health in both zero-shot and few-shot learning settings. While prior work has shown significant differences in performance for these models on standard summarization tasks, our results indicate that with as few as 10 labeled examples there is no statistically significant difference in summary quality, suggesting the need for more abstractive benchmark collections when determining state-of-the-art.

## introduction since its inception @xcite , automatic summarization has focused on summarizing documents either in a generic way -conveying the main points of the document to any reader regardless of their information need -or in a task-specific way -distilling the important points of the document with respect to a specific information need such as a question or topic statement @xcite . in the latter case, the selection of the most salient points in the document (i.e., content selection) as well as the expression of those points (i.e., surface realization) must be explicitly conditioned on a user-given natural language context statement, such as a question or topic of interest. in this setting, a single passage may be summarized in different ways depending on the context description. consequently, obtaining reference summaries is often time-or cost-prohibitive, particularly when dealing with specialized domains such as healthcare. the document understanding conference has explored topic-driven summarization (duc) and its successor, the text analysis conference (tac), which both ran community evaluations of topic-or question-based summarization. specifically, participants were asked to develop automatic summarization approaches for generating single-or multi-document summaries that summarized a set of documents with respect to a given topic description or question, as shown in figure 1 . human assessors manually judged submitted summaries. in this work, we revisit the multi-document topic-driven abstractive summarization datasets produced from @xcite @xcite well as question-driven summarization from consumer health. because these datasets are relatively small (approximately 45 topics each), we explore modern transformer-based models' performance in the zero-shot and few-shot (10 examples) learning settings. specifically, we explore the quality of multi-document abstractive summarization generated by t5 @xcite , bart @xcite , and pegasus @xcite . this work is licensed under a creative commons attribution 4.0 international license. license details: http: //creativecommons.org/licenses/by/4.0/ . question: what became of the cast and others related to the "seinfeld" tv series after it ended? what actions were taken by others in response to the show's closing? summary: after "seinfeld" ended, jerry seinfeld did a stand-up tour, a 10-show broadway run, and an hbo special, "i'm telling you this for the last time". he ## background recent work has indicated that transfer learning (pre-training a model on data-rich tasks before fine-tuning it on a downstream task) obtains remarkable performance on many natural language processing tasks @xcite @xcite . the most successful models are obtained through self-supervised pre-training with massive datasets to obtain transferable knowledge for new tasks (i.e., fine-tuning) with less abundant data @xcite @xcite . more recently, research has indicated that these models can generate language conditioned on a user-given prompt or context. for example, this prompt can guide the model's content selection towards a particular topic @xcite or inform surface realization for a specific task @xcite . in @xcite , the authors condition an extractive transformer using "control codes" to specify the position, importance, and diversity of the sentences in the source text. in this work, we adapt this paradigm to train and evaluate bart, t5, and pegasus for abstractive multi-document summarization. although zero-shot learning (zsl) has received considerable attention in the image processing community, there has been comparatively little work on zero-shot learning specifically for summarization: duan et al. (2019) explore zero-shot learning for cross-lingual sentence summarization and @xcite explored zero-shot abstractive summaries of five-sentence stories. we extend these works by evaluating zero-shot and few-shot learning for multi-document abstractive summarization. ## models in this work, we compare three of the most prominent conditional language generation models: t5, bart, and pegasus. to facilitate comparison, for each model we chose the variant with the most similar architecture (such that each consists of 12 transformer layers and a similar number of learnable parameters). each model is pre-trained with unique strategies as described below. bart (bidirectional and auto-regressive transformers) is pre-trained on document rotation, sentence permutation, text-infilling, and token masking and deletion objectives @xcite . in our experiments, we used bart-large. t5 (text-to-text transfer transformer) is pre-trained on several unsupervised and supervised objectives, such as token and span masking, as well as translation, classification, reading comprehension, and summarization. importantly, each objective is treated as a language-generation task, where the model is conditioned to generate the correct output based on a textual prompt included in the input sequence @xcite . in this work, we used t5-base. pegasus (pre-training with extracted gap-sentences for abstractive summarization sequence-tosequence) was specifically designed for abstractive summarization and is pre-trained with a self-supervised gap-sentence-generation objective @xcite . in this task, entire sentences are masked from the source document, concatenated, and used as the target "summary". we used pegasus-base in our experiments. ## experiments we evaluated t5, bart, and pegasus in zero-shot (zsl) and few-shot (fsl) learning settings on four datasets. summary quality was measured using rouge-1, rouge-2, and rouge-l 1 -scores @xcite ; bleu-4 @xcite ; and repetition rate (in unigrams). implementation details are provided in appendix a. ## conclusion we evaluated the summarization quality produced by three state-of-the-art transformers: bart, t5, and pegasus on four challenging summarization dataset in both zero-shot and few-shot learning settings. our results indicate that, while there are statistically significant differences between the models in zero-shot settings, after few-shot learning with as few as 10 examples, there is little discernible difference between them. this suggests that while large improvements have been made on standard single-document benchmarks, highly abstractive multi-document summarization remains challenging. question: what are the causes of childhood obesity? context: overweight and obesity (risk factors): there are many risk factors for overweight and obesity. some risk factors can be changed, such as unhealthy lifestyle habits and environments. other risk factors, such as age, family history and genetics, race and ethnicity, and sex, cannot be changed. heathy lifestyle changes can decrease your risk for developing overweight and obesity. unhealthy lifestyle habits lack of physical activity, unhealthy eating patterns, not enough sleep, and high amounts of stress can increase your risk for overweight and obesity. lack of physical activity lack of physical activity due to high amounts of tv, computer, videogame or other screen usage has been associated with a high body mass index. reference summary: there are many factors that contribute to increasing your child's risk of becoming overweight. regularly eating high calorie foods can cause your child to gain weight. lack of physical exercise can contribute to childhood obesity. if your child comes from a family of overweight people, he or she may be more likely to put on weight. personal, parental, and family stress can also increase a child's risk of obesity.