U ni S umm and S umm Z oo: Unified Model and Diverse Benchmark for Few-Shot Summarization (2023)

Abstract: The high annotation costs and diverse demands of various summarization tasks motivate the development of few-shot summarization. However, despite the emergence of many summarization tasks and datasets, the current training paradigm for few-shot summarization systems ignores potentially shareable knowledge in heterogeneous datasets. To this end, we propose UniSumm, a unified few-shot summarization model pre-trained with multiple summarization tasks and can be prefix-tuned to excel at any few-shot summarization task. Meanwhile, to better evaluate few-shot summarizers, under the principles of diversity and robustness, we assemble and release a new benchmark SummZoo. It consists of 8 summarization tasks with multiple sets of few-shot samples for each task, covering diverse domains. Experimental results and analysis show that UniSumm outperforms strong baselines by a large margin across all sub-tasks in SummZoo under both automatic and human evaluations and achieves comparable results in human evaluation compared with a GPT-3.5 model.

## introduction there has been a recent surge of interest in summarizers based on large pre-trained language models (plms) @xcite @xcite @xcite , where various summarization tasks (the term task later in this paper refers to a specific summarization task, e.g., query-focused meeting summarization, which is usually associated with a corresponding dataset, e.g., qmsum, unless otherwise specified.) have been proposed to meet different practical demands, such as comprehending different inputs (e.g., news @xcite and dialogue @xcite ) and generating different outputs (e.g., headlines (zhang and * yulong chen completed this work during his internship at microsoft. how to summarize their talk? ## method following @xcite , the task of fewshot text summarization is defined as follows. for an unseen target summarization task u, few-shot text summarization is to generate a summary y , given an input text x, by learning from a limited number k (@xmath0, with the help of general knowledge k. the overall framework of unisumm is shown in figure 2 . it consists of 2 phases: 1) learning general knowledge by multi-task pre-training on existing summarization datasets ( § 3.1) and; 2) learning target task knowledge by prefix-tuning on each target few-shot summarization dataset ( § 3.2). ## the summzoo benchmark summzoo is sourced from existing summarization benchmark based on the principles of diversity and robustness, where we assemble each dataset into few-shot evaluation settings. diversity of tasks as a major goal, we ensure that summzoo can include a diversity of different summarization tasks, covering multiple domains, text styles and compression ratios. thus, we carefully select 8 summarization tasks including monologue/dialogue texts and single/multi-document summarization tasks. their domains also span an assorted set such as news, scientific papers, instructions, online forums and meetings. ## human evaluation in human evaluation, unisumm outperforms pegasus and bart-pt on all datasets regarding all dimensions, achieving a higher fluency score than gold summaries on qmsum and a comparable score on multinews and wikihow, suggesting that unisumm can generate very fluent sentences which can be comparable with human annotated summaries. a challenge of qmsum is that models are asked to generate summaries focusing on the input queries. thus, relevance is a very important metric for this task. however, relevance sees very low score for pegasus (3.27) and bart-pt (2.80), suggesting they are weak in extracting relevant information based on user queries. in contrast, unisumm achieves a higher score (3.97). text-davinci-002 also performs very well on this task, even outperforming the gold summaries on fluency, but unisumm still achieves comparable results with limited training samples and much lower cost. on multinews, since text-davinci-002 is only input with 1-shot summary as icl example due to length limitation, although it can generate very fluent (4.97) and coherent (4.73) summaries, it is less preferred by human annotators w.r.t. consistency and relevance. unisumm still outperforms other systems and only loses to gold summaries on this two metrics. similar results are also observed on wikihow, where text-davinci-002 tends to generate very long summaries, which can con-tain some hallucination and less important content, and unisumm shows comparable performance on consistency and relevance. we show case studies and their analysis, including an error case where unisumm fails, in appendix f. ## conclusion we introduced unisumm, a novel few-shot summarization system that can be easily prefix-tuned to excel at and generalize on a diversity of summarization tasks. we propose to combine multitask learning and prefix-tuning by jointly training the prefixes and the summarizer on multiple existing summarization datasets. by only tuning the prefix parameters, unisumm shows superior performance over strong baseline systems, yielding fluent and faithful summaries across tasks. in addition, we assembled and released a new benchmark, summzoo, for fairly and effectively evaluating few-shot summarization models. it covers an assorted set of summarization tasks and provides multiple few-shot sets for a more robust and fairer comparison.