USB : A Unified Summarization Benchmark Across Tasks and Domains (2023)

Abstract: While the NLP community has produced numerous summarization benchmarks, none provide the rich annotations required to simultaneously address many important problems related to control and reliability. We introduce a Wikipedia-derived benchmark, complemented by a rich set of crowd-sourced annotations, that supports 8 interrelated tasks: (i) extractive summarization; (ii) abstractive summarization; (iii) topic-based summarization; (iv) compressing selected sentences into a one-line summary; (v) surfacing evidence for a summary sentence; (vi) predicting the factual accuracy of a summary sentence; (vii) identifying unsubstantiated spans in a summary sentence; (viii) correcting factual errors in summaries. We compare various methods on this benchmark and discover that on multiple tasks, moderately-sized fine-tuned models consistently outperform much larger few-shot prompted language models. For factuality-related tasks, we also evaluate existing heuristics to create training data and find that training on them results in worse performance than training on 20\times less human-labeled data. Our articles draw from 6 domains, facilitating cross-domain analysis. On some tasks, the amount of training data matters more than the domain where it comes from, while for other tasks training specifically on data from the target domain, even if limited, is more beneficial.

## introduction automatic text summarization has been an important, active research sub-area in nlp for over two decades @xcite @xcite . numerous summarization benchmarks have been proposed to facilitate the development of summarization methods @xcite @xcite . however, the majority of previous work has primarily focused on evaluating the models' ability to generate summaries similar to reference summaries, neglecting key auxiliary properties of text summarization systems. recent research has highlighted the importance of addressing additional aspects in text summarization. these aspects include the ability to steer summaries by controlling its focus on a topic or on specific parts of the source text @xcite . furthermore, there is an increasing emphasis on ensuring factual correctness and implementing mechanisms to eliminate factual errors from model outputs @xcite . similarly, to foster trust in the outputs, it is desirable for summarization systems to present evidence from sources that corroborate the generated summaries. as models have improved in generating coherent and readable summaries @xcite , these auxiliary considerations have gained importance. aligning summaries with user requirements and ensuring sufficient factual support are critical frontiers in summarization research. the current summarization benchmarks fail to provide a comprehensive evaluation of model capabilities across various summarization tasks, encompassing properties such as factuality and controllability. in this work, we introduce usb, a comprehensive benchmark for text summarization that supports eight auxiliary tasks. the benchmark includes labeled datasets with high-quality human annotations collected from diverse documents across six domains. to create the benchmark, we sampled wikipedia articles from various categories, such as people, organizations, and events. we utilized the introductory section of the articles as a reference summary and the remaining content as the source text, resulting in imperfect source-"summary" pairs. human annotators then searched for evidence to support each summary sentence. if evidence was lacking, corresponding spans or entire sentences figure 1 : a schematic of our dataset, annotations, and the supported tasks. the example shown (abridged) displays the edits made by a human annotator on the initial candidate summary (deletions in red with strike-through; additions in green). every summary sentence is supported by one or more evidence sentences highlighted in blue. we offer the first human-labeled training datasets for various summarization tasks, including evidence extraction and identifying spans in summaries without supporting evidence. these datasets enable the training and evaluation of models specifically for these crucial aspects. we benchmark the performance of several models such as instructiontuned encoder-decoder models and llms on our tasks, including both fine-tuning and well as fewshot prompting based approaches. notably, we found that fine-tuning even small models (fewer than a billion parameters) substantially outperforms few-shot prompting of much larger open-source and private large language models. prior efforts have relied on heuristics to generate synthetic training data for certain tasks included in our benchmark. for instance, a common heuristic employed is lexical overlap to identify and extract supporting evidence @xcite . similarly, artificial factual errors have been introduced into summaries to train models for factuality classification or correction @xcite . although such automatic approaches enable easy creation of large training datasets, heuristically derived annotations are typically noisy compared to human annotations. our findings demonstrate that models trained on minimal amount of human-labeled data outperform those trained on heuristically generated labeled datasets, even when the latter are 20× larger. a common challenge to real-world adoption of models is their use in resource-poor domains where one does not have access to abundant labeled training data. we compare how the size of available training data matters vis-a-vis its domain for different summarization tasks. we found that for tasks related to factual correctness of summaries, the amount of training data matters more than its domain; but for other tasks having domainspecific training data matters more. our benchmark is explicitly segmented into 6 domains based on wikipedia categories, and hence provides a natural test-bed for such domain transfer studies. ## dataset curation to create the usb benchmark, we first collected a set of manual annotations on wikipedia articles. we then used the collected annotations to create labeled data for the benchmark tasks. in this section we describe the process of collecting these manual annotations. we consider the text in a wikipedia article overview (leading) section as the target summary s, and the rest of the article as d. in wellwritten articles, the overview section (s) provides a broad summary of the article, and the rest of the article (d) provides specifics. hence, the content in s which highlights parts of d can be effectively considered its summary. however, for s to be a valid summary of d, we need to remove contents within it that mention new information that is not present in d and cannot be inferred from it. we recruited annotators and asked them to execute the following tasks: (1) find and annotate evidence in d for each summary sentence of s, and; (2) delete parts of s that are not supported by d. this yields a document-summary pair where the summary is fully supported by the document, and the supporting evidence is explicitly marked. we provide a detailed description of our data creation process below. ## task definitions we derived labeled datasets for tasks using the collected annotations. the resulting benchmark consists of the following 8 tasks: extractive summarization (ext): given the full document as input, extract all important sentences that it contains. we define the ideal "reference" extractive summary as the set of all source sentences marked as evidence for the summary. abstractive summarization (abs): generate a multi-sentence summary of the source document by not just simply selecting important content, but also rephrasing it for succinctness and coherence. the ground truth is the full-length summary created after the annotators' edits. factuality classification (fac): predict if a summary sentence is factually correct and sufficiently supported by the information present in the source. we create labeled data by assigning non-factual and factual labels to the before and after versions of each edited summary sentence, with the marked evidence as source context fed in the input. ## dataset overview and statistics the usb is a benchmark comprising 6 different domains with a varying number of instances in each ( ## benchmarking different models we run a suite of models on all tasks in our benchmark and present the results in for unsupported span prediction, we measure the f1 score based on bio tagging format @xcite . for the remaining tasks we use standard binary classification metrics. for the classification/span prediction tasks in our benchmark, we fine-tune roberta-large @xcite ; we run large language models including publicly released models (for research purposes) such as llama @xcite and vicuna @xcite , and closed models such as openai's gpt-3.5-turbo foot_4 , i.e., chatgpt. for tasks where the full document is fed as input, we use 4 examples for few-shot prompting owing to limitations in the maximum feasible sequence length for these models, while for the rest we use 16 examples (for details, see the appendix). chatgpt consistently outperformed vicuna-13b and llama-13b on all tasks except fixing factuality. this is because for the fixing factuality task, chatgpt almost always adds new unnecessary information to the summary, even after prompting it to not do that. compared to chatgpt, finetuned models perform better on every task. the performance difference is largest for factuality-based tasks such as unsupported span prediction, evidence extraction, and fixing factuality. chatgpt does comparatively well on tasks that involve generating summaries. since automatic metrics for measuring summary quality like rouge @xcite do not necessarily mirror human preference @xcite , we conducted human evaluation of the generated summaries in the comp, abs and topic tasks. we collect ratings for summaries generated by flan-t5 and chatgpt for 50 randomly selected documents from the test set, using a questionnaire (see the appendix for more details). we found that 3 ). this suggests that while fine-tuned models produce summaries closer to the ground truth in the dataset (thus achieving high rouge), humans may find the summaries of fewshot prompted llms better. for example, in the topic-based summarization task, while flan-t5-xl produces summaries with an average length of 46.3 words, chatgpt generates summaries with an average length of 110.2 words. the ground truth summaries for that task are 36.9 words long on average, which is more closely matched by flan-t5-xl, but the much longer summaries of chat-gpt are considered better by human annotators as reflected in the human ratings ( for the fixing factuality (fix) task, we compare the fixed summaries generated by flan-t5-xl and chatgpt, asking which of them (i) remove more factual errors; (ii) mistakenly remove more correct information; (iii) add more new information; to the initially provided incorrect summary. we found that while chatgpt removes more factual errors from summaries than flan-t5, it often does so by removing lots of (even factual) information altogether, and replacing it with new content to effectively make a new different summary ( ## out-of-domain performance on tasks we next evaluate the performance of fine-tuned models when tested on a domain different from what they were trained on. our benchmark has training data from 4 domains (i.e. excluding landmarks and newspapers), with different amounts of labeled data for each. to control for training set size, we randomly subsample annotated documents for each domain to isolate 40, 19, and 38 documents for train, validation and test splits. these sizes of the splits were chosen to match the smallest of the 4 domains i.e. companies ( we train and evaluate flan-t5 large models on different domains and plot average scores for all tasks training and test domain pair in figure 2 . models trained on the same domain as the test domain perform best or negligibly close to it. but across test domains, the best out-of-domain trained model @xmath0, showing respectable average out-of-domain performance. going by the in-domain performance of models trained on equal amounts of data, the biographies domain is the easiest and the disasters domain is the most difficult. one distinction between the disasters domain and others which might explain its difficulty is that it deals with summarizing an event rather than an entity. for each task in our benchmark, we investigate whether having access to a large training dataset (irrespective of domain) is more important than having training data from the test domain. we use the test splits of 3 domains (companies, disasters, and schools), and on each of them we evaluate 3 different models trained on: (1) the training split of the same domain; (2) the training split of the ## comparison with heuristic annotations for some tasks in our benchmark, past works have used heuristics to create large labeled training data sets as an alternative to collecting manual annotations @xcite @xcite . we use such proposed heuristics to train models and compare them with models trained on high-quality, human annotated data. we conduct experiments on the evidence extraction, factuality classification and fixing factuality tasks. because the primary advantage of heuristic-generated training sets is their size, we also assess how smaller human-labeled training sets fare in comparison. for the evidence extraction task, we use lexical overlap as a proxy to derive "reference" evidence alignments. for example, we select the source sentence with the highest rouge-l score with a summary sentence as its evidence, as outlined in @xcite . we also create a training set variant where entity overlap is used instead of evidence extraction f1 superpal @xcite 53.8 rouge @xcite 40.9 entity overlap 47.0 rouge-l to derive references. finally, we use superpal @xcite as an out-of-the-box solution to predict evidence labels for our dataset's summaries, and then use them for model training. to train models to detect or fix factual errors, we artificially introduce errors into summaries to be used as exemplars. we do this via transformations such as swapping entities, numbers, pronouns, introducing negation, and so on, inspired by prior work @xcite . to generate diverse errors and hallucinations, we follow balachandran et al. (2022); we mask parts of the summary out and then use a language model to infill these with (mostly unsupported) information. we train models for 3 tasks on both heuristicallygenerated and manually annotated training datasets, and evaluate them on clean human-labeled test sets. training on human-annotated data performs better than all heuristic-based alternatives across all tasks ( ## related work the tasks in our benchmark have been studied in prior work to varying degrees. the greatest amount of attention has gone to the tasks of extractive summarization @xcite @xcite , and abstractive summarization @xcite @xcite @xcite . there exist plenty of datasets for abstractive summarization @xcite @xcite . however, many of them were created heuristically, with "targets" being automatically extracted via rules from documents pulled from the web. this can lead to poor quality reference summaries @xcite , and training on them can yield models prone to generating hallucinations @xcite . by contrast, we use manual annotation to ensure that summaries are fully supported by sources, resulting in a high quality abstractive summarization dataset. past works for predicting factual correctness of summaries incorporate question-answering models and natural language inference methods @xcite @xcite , or use synthetically introduced factual errors @xcite to train models. in contrast, the usb benchmark introduces a highquality manually annotated dataset for predicting factual correctness. for the task of editing summaries to fix factual errors, datasets with both synthetic and model-generated errors have been created @xcite . the task of unsupported span prediction is akin to detecting hallucinated content in generated summaries, and to the best of our knowledge, no labeled dataset exists for this task. for extracting evidence for a summary, past works have used lexical overlap based heuristics @xcite . a manually annotated dataset for the task was introduced by @xcite , albeit our work provides a substantially larger manually annotated dataset. similarly, for multi-sentence compres-sion we introduce a much larger manually labeled dataset than prior works @xcite . prior research has mostly approached topic based summarization by adopting a predefined set of topics @xcite @xcite . however, we did not restrict the set of topics in our dataset, resulting in a long tail of (potentially challenging) rare topics. ## conclusion we introduced the usb benchmark comprising tasks to measure model performance across different text summarization sub-tasks. we showed that fine-tuned smaller models outperform few-shot prompting of much larger llms by a large margin on tasks related to appraising the factuality of summaries. we studied how fine-tuned summarization models perform on out-of-domain data, and identified several tasks where the training dataset size is more important than its domain. finally, we showed that rather than training models on large volumes of heuristically labeled data, one can get better performance by creating a much smaller (≈ 20× smaller) manually labeled training set instead. the resultant usb benchmark permits the training of models for useful tasks such as extracting evidence for a summary, correcting factual errors in it, and generating summaries focused on specific topics. our hope is that this benchmark spurs further research on these tasks and will serve as a barometer for progress in them. wikipedia articles are written and formatted. the dataset's reliance on wikipedia articles as the primary source of data might not adequately represent the nuances and challenges encountered in different domains or sources. one prominent example is conversations which are frequently used in summarization research but are not represented in the benchmark. similarly, a model's ability to detect errors/hallucinations in summaries in the benchmark may not necessarily reflect its ability to detect errors more broadly in summaries generated by a variety of models. while the benchmark dataset was annotated by human annotators, it is important to acknowledge the possibility of annotation errors or inconsistencies. despite efforts to ensure high-quality annotations, the presence of errors should be taken into account when interpreting the results. human annotation is subjective by nature, and different annotators may have varying interpretations in some situations, e.g., deciding whether a fact in the summary requires explicit evidence or should be presumed as common knowledge. for the tasks of abstractive summarization (abs), multi-sentence compression (comp), and topic-based summarization (topic), we collected annotations for 50 pairs of summaries, with 3 annotators rating each pair. for these 3 tasks, we did not screen workers based on qualification tasks since evaluating overall summary quality is a subjective task and it is better to have a diverse opinion from a large population, rather than a small set of manually selected people. evaluating model outputs for the fixing factuality (fix) task is a more difficult but objective job. the increased difficulty comes from the need to carefully note the edits made by the models on the original incorrect summary and then decide on the factual validity and necessity of each edit. so we screened annotators via a qualification task on mechanical turk and selected 2 annotators to conduct the human evaluation for this specific task. each pair of model outputs was rated by both annotators.