GUMS um: Multi-Genre Data and Evaluation for E nglish Abstractive Summarization (2023)

Abstract: Automatic summarization with pre-trained language models has led to impressively fluent results, but is prone to ‘hallucinations’, low performance on non-news genres, and outputs which are not exactly summaries. Targeting ACL 2023’s ‘Reality Check’ theme, we present GUMSum, a small but carefully crafted dataset of English summaries in 12 written and spoken genres for evaluation of abstractive summarization. Summaries are highly constrained, focusing on substitutive potential, factuality, and faithfulness. We present guidelines and evaluate human agreement as well as subjective judgments on recent system outputs, comparing general-domain untuned approaches, a fine-tuned one, and a prompt-based approach, to human performance. Results show that while GPT3 achieves impressive scores, it still underperforms humans, with varying quality across genres. Human judgments reveal different types of errors in supervised, prompted, and human-generated summaries, shedding light on the challenges of producing a good summary.

## introduction recent advances in supervised summarization models as well as prompt-based approaches using large pre-trained language models have led to substantial improvements in summary fluency, with prompt-based outputs now surpassing supervised approaches in human evaluation @xcite . at the same time, researchers in the field repeatedly note that the most commonly used datasets, such as cnn/dailymail (cnn/dm, @xcite and extreme summarization @xcite , which are large-scale 'found' datasets not designed to facilitate high quality summarization, are problematic, and in many cases contain texts which are not summaries, are incomplete or unfaithful to the texts they relate to, add information not present in texts, or any combination of the above @xcite . existing datasets are also limited to mainly newswire text (cf. @xcite , which is a fraction of extant genres in general and on the web. the main contributions of this paper are in providing and evaluating a very genre-diverse dataset and guidelines for carefully crafted, rather than 'found' summaries, which follow the same design across text types. building on the ud english gum treebank @xcite , which contains 213 spoken and written texts balanced across 12 different genres, our summaries target three goals: 1) to be substitutive (i.e. informative, functioning as a substitute for reading a text, cf. @xcite rather than indicative (e.g. 'clickbait' designed to attract readership); 2) to be faithful to the text, adhering to original formulations wherever possible; 3) to be hallucination-free, meaning summaries make a strong effort not to add any information (even if it is likely to be true), mentioning only entities and events actually contained in the text, thereby preventing typical errors associated with datasets such as xsum @xcite . instructions on obtaining the dataset and responses from the human evaluation study as well as evaluation code can be found at https://github.com/janetlauyeung/ gumsum4eval . foot_0 ## related work the problem of mitigating factuality and faithfulness issues in natural language generation (nlg) has recently received considerable attention, with studies proposing auxiliary tasks using the multi-task learning approach to constrain models, such as overlapping entities @xcite , encoding of predicate triples from source documents @xcite or encouraging systems to incorporate or copy entities from source documents @xcite . in addition, @xcite present a thorough investigation of factual errors in summarization and propose a taxonomy of error types with a focus on entity and predication errors, while @xcite examine types of accuracy errors made by neural systems and contrast them with human errors. these papers share concerns about the nature of widely used datasets for english, such as xsum and cnn/dm, but are limited by the lack of evaluation data specifically targeting genre-diverse texts with high-quality summaries: ones which ideally maximize faithfulness, rule out hallucinations, and follow consistent guidelines for what constitutes a summary. although there are some non-news single-document summarization datasets covering reddit @xcite and podcast data @xcite ), text types are still quite limited and data is often not publicly available @xcite . this motivates our work to create openaccess, multi-genre data with consistent guidelines across text types. ## dataset contents gumsum covers the 213 documents (amounting to ∼200k tokens) from the 12-genre ud english gum corpus @xcite ; specifically gum v9), which provides gold syntax trees, entity types, coreference resolution, and discourse parses for the data. for this paper, we added summaries to each document in the corpus, by the authors and students in a computational linguistics course as part of a class-based project, 2 guided by general and genre-specific instructions. although the range of ∼20 human summarizers is broad as a result, we defined guidelines to constrain summaries and ensure they are maximally 'realitychecked', i.e. faithful and factual, as evaluated below. documents vary in length, ranging between 167 and 1,878 tokens (@xmath0, @xmath1.6), and cover the genres in what these qualities mean varies depending on the summary's objective: whether it is domain-specific or general, indicative (enticing readers to read the text) or informative (aiming to substitute reading it, nenkova and mckeown 2011) etc. gumsum's summaries explicitly target a domain-general, substitutive, maximally concise format, which is therefore constrained to: [1] have at most one sentence / 380 characters foot_1 [2] have the goal of replacing reading the text [3] give participants/time/place/manner of events [4] form a sentence rather than a fragment [5] omit distracting information [6] avoid entities or information not present in the text, even if we are fairly sure it is true [7] reject synonyms for words in the text for instance, the summary in (1) for a story involving 'robbers plundering a vault' follows guidelines by providing a declarative-sentence (criteria [1], [4]), synopsis of events, participants (exactly five robbers), time (a date) and place (poughkeepsie) ([3]), as well as additional details (exact name of the bank, mode of escape). (2) is underspecified (we do not know when or where the event occurred, criterion [3]). (3) paraphrases the robbers' escape by introducing an entity not in the original text (uncaught by police, violating [6]), and substitutes 'robbed' for 'plundered', a near synonym but a deviation from the original text's style @xcite ). bank robbers plundered a vault and escaped. bank robbers who robbed a bank in poughkeepsie were never caught by police. although these examples illustrate newswire language, gumsum covers very different spoken and written text types as well: the summary in (4) follows the guidelines by not mentioning that the discussion is on reddit ([6], the interlocutors are simply 'people'), since reddit is not mentioned. similarly, while zack snyder's film batman v superman: dawn of justice is most likely being discussed, it is not named explicitly, leading to the formulation 'snyder's movie about batman and superman'. in (5), the summary focuses on conveying events which happen over the course of a vlog, but again, the unmentioned word 'vlog' is avoided, while specific details about the participants and circumstances (people, but also the type of class) are prioritized. summaries are thus highly constrained to remain faithful and avoid even minor potential hallucinations, such as completing the title of a film. for more on genre-specific guidelines and examples, see appendix a. ## evaluation automatic evaluation to evaluate how well current neural approaches produce 'reality-checked' summaries approaching the ones in gumsum, we obtain system outputs from two recent supervised systems, brio @xcite and simcls @xcite , as well as prompt-based outputs using a gpt3 model @xcite , gpt3-text-davinci-002 (gpt3-dv2), with the prompt 'summarize the text above in one sentence.'. we chose system models trained on the xsum dataset, since it has one-sentence summaries more in line with the gumsum data. however, because systems have never seen data in many of gum-sum's genres, we also add an additional experiment in which we fine-tune the higher-scoring supervised system, i.e. brio's trained-model on xsum for generation, by continuing training it on the 165 documents in the ud treebank's train set of the underlying gum v9 corpus (brio-ft in it is an established finding however that metrics do not tell the whole story @xcite @xcite . in fact, we regularly observe hallucinations, especially in xsum-trained systems, such as prefixing generic leads (e.g. 'in our series of letters from british journalists ...', when there are no journalists involved) or inserting entities and events not mentioned in the text. we thus conduct a human evaluation of system outputs below, focusing on substituitivity, hallucinations, and faithfulness, and more importantly, apply the same evaluation criteria to the human-written summaries for a more targeted evaluation, as advocated by @xcite . ## conclusion the dataset and guidelines introduced in this paper make a step towards consistent and constrained multi-genre evaluation of factual summarization. our results show that domain-general summarization is still hampered by serious reliability and factuality problems, which may only become apparent when confronted with a dataset with strict 'reality check' constraints and diverse text types. even small amounts of such data can be used to fine-tune pre-trained systems, with measurable improvements for system outputs. the human evaluation study also revealed that pre-trained systems are bad at delivering substitutive summaries, perhaps because, as pointed out in @xcite , "summarisation datasets should contain summaries," but often they do not. meanwhile, human identification of possibly more minor hallucinations in human-written summaries also suggests that more work is needed in delimiting what a 'reality check' for summaries should include.