SQ u ALITY : Building a Long-Document Summarization Dataset the Hard Way (2022)

Abstract: Summarization datasets are often assembled either by scraping naturally occurring public-domain summaries—which are nearly always in difficult-to-work-with technical domains—or by using approximate heuristics to extract them from everyday text—which frequently yields unfaithful summaries. In this work, we turn to a slower but more straightforward approach to developing summarization benchmark data: We hire highly-qualified contractors to read stories and write original summaries from scratch. To amortize reading time, we collect five summaries per document, with the first giving an overview and the subsequent four addressing specific questions. We use this protocol to collect SQuALITY, a dataset of question-focused summaries built on the same public-domain short stories as the multiple-choice dataset QuALITY (Pang et al., 2021). Experiments with state-of-the-art summarization systems show that our dataset is challenging and that existing automatic evaluation metrics are weak indicators of quality.

## introduction research on automatic text summarization generally requires adequate benchmark datasets. existing datasets in this area often have issues that seriously limit their usability: for instance, summaries from the popular scraped benchmark summarization dataset cnn/dailymail @xcite contain html artifacts, links to other news articles, and other types of noise @xcite . a common approach to creating summarization datasets is to develop heuristics to extract pseudosummaries from existing texts. while scraped summaries can be cleaned of noise, these heuristics can lead to more fundamental data artifacts. for example, the xsum dataset @xcite was one writer rst creates four questions, with an additional xed question used for every story. then, four writers each write summaries answering the ve questions. next, each writer ranks the other three summaries for each question and provides written feedback. finally, we aggregate ranks and award bonuses to incentivize high-quality summaries. between collection rounds, writers review the feedback they received. another approach to creating summarization datasets relies on serendipity in nding naturally occurring summaries. for example, the arxiv and pubmed datasets @xcite use the abstracts of scientic papers as summaries of the papers. bigpatent @xcite and govre-port @xcite use expert-written summaries that come with patent lings and government reports, respectively. while these summaries are likely high-quality, the domain of the data poses a signicant challenge for system evaluation: automatic evaluation metrics for summarization are unreliable @xcite , but the summaries are too technical and jargonistic for non-specialist human raters to evaluate reliably. because we rely on chance in nding these summaries, we are beholden to whatever domain they come from, rather than the domain we are interested in. relying on nding and scraping summarization data is also problematic in that, often, the found data is proprietary and not freely distributable. for example, many researchers and organizations are unwilling to host or distribute the cnn/dailymail dataset, 1 despite it being one of the most popular summarization datasets to experiment on. similarly, several recent summarization datasets built on data such as scientic journal papers @xcite or sparknotes book summaries @xcite have never been made available to researchers. the dataset creators instead ask potential users to re-scrape them, which can be a serious obstacle to reproducibility. in this work, we propose a crowdsourcing protocol for collecting original summaries free of these issues. crowdsourcing summaries has been underexplored because straightforward approaches for doing so are quite labor-intensive. while our protocol is still fairly demanding, we structure it in a way that makes the cost per summary more tractable (∼$6/summary) while also including incentives and checks to ensure the summaries are high-quality. the protocol does not rely on nding naturally occurring summaries and is agnostic to the input documents used, so we are free to choose the input documents we want to summarize. we use short stories from project gutenberg to avoid the aforementioned domain and licensing issues. we use this protocol to collect squality 2 (summary-format question answering with long input texts, yes!), a dataset for question-focused abstractive summarization of short stories. squal-ity summaries are created by having trained writers read short stories, then ask questions about different aspects of the story. the writers then answer 1 see discussion here. 2 named because it uses many of the same stories as the multiple choice qa dataset @xcite . the questions by writing summaries focusing on that aspect. each question is answered by four different annotators, who then review each other's work to ensure the data is high-quality. in total, squality consists of 100 stories, 500 questions, and 2000 summaries. 3 overall, we make the following contributions: ## related work story summarization a common focus of summarization research is on stories and narratives. booksum @xcite consists of public domain books and summaries of those books, chapters, and paragraphs. similarly, @xcite propose a dataset for summarizing chapters of public domain books. both of these datasets use summaries scraped from popular study guide websites such as sparknotes, apparently without an overt license, and thus the datasets cannot be legally distributed. summscreen @xcite consists of fan-written transcripts of tv title: pick a crime ( https://www.gutenberg.org/ebooks/51656 ) q: what is the cpa and what does it do? the crime prevention association is an organization that stops crime. instead of capturing criminals, the goal of the association is to prevent the crime from ever happening. they implement thousands of crime-prevention methods and devices. there are many amateur cops who constantly follow criminals around in hopes of catching them in the act so that they may be hailed a hero and... the cpa is crime prevention organization. it ghts crime by all means and reduces its rates to a very small level. they put microphones and detectors everywhere to hear the conspiracies. they place robots as bartenders to control the level of alcohol in visitors to prevent them being drunk. they make all the women learn self-defense. the organization's made crime almost impossible... the cpa, crime prevention association, is a system that detects different kinds of crimes and prevents them from happening. thousands of robots and devices make crimes impossible. the association will not punish any crime, instead, the criminal will be send to a cpa hospital for some treatments that will result in getting the best jobs. the cpa also hands out id cards that states one's... the cpa is meant to prevent crime and not punish crime. it stands for crime prevention association. the cpa organization has made crime nearly impossible through various methods of surveillance and intelligence gathering. the crime was not punished by the cpa but addressed by sending the person to a hospital for expensive treatment to correct and remove the deviance from the person's... ## dataset construction source documents our considerations in selecting a corpus of documents for which to collect summaries are: (1) the documents are long, as document-level tasks are more challenging than paragraph-level ones; (2) the documents can support several substantive summaries, as we will collect multiple summaries per document for cost-efciency; (3) the documents have a permissive license so they can be easily distributed; (4) the documents are lay-accessible, such that the average college-educated english-uent speaker can both understand them and condently evaluate the correctness of summaries derived from them. we use short stories from project gutenberg as they meet all of these desiderata. 4 specically, we use a collection of science ction short stories written in @xcite and are between 3000 to 6000 words long. many of the stories used are also included in the quality @xcite dataset, and we coordinate with the quality creators such that stories that appear in both datasets are assigned to the same split. we use the same preprocessing for the stories as used in quality. writing for writers to create accurate and highquality summaries, they need to read the entire story, which takes 20-40 minutes. rather than asking writers to create one summary per story read, we instead collect multiple summaries per story to amortize the cost of reading across summaries. we solicit multiple summaries by having writers ask questions about different aspects of the story, leading us to create a qfs dataset. we start each crowdsourcing round by asking writers to read the story and then create questions satisfying two criteria: (1) questions should require the whole or multiple parts of the story to answer, as opposed to a single sentence or paragraph; (2) to minimize disagreements in evaluation, writers should avoid questions that require speculating substantially beyond the literal text of the story when interpreting themes or symbolism. to assist writers in creating questions satisfying these properties, we provide a list of question templates we expect will meet these properties in most cases, shown in appendix a.1. writers can also write story-specic questions not based on any of these templates so long as they follow the criteria. for each story, we assign one worker to create four questions. the questions are then answered by four writers, including the original question writer. each writer also creates a general story summary, framed as answering the question "what is the plot of the story?", for a total of ve questions per story. responses are required to be 75-500 words long, to avoid copying the text of the story verbatim, and to draw on different parts of the story as much as possible. writers report that this step takes 40-120 minutes, including time reading the story. ## squality we present examples from squality in data size and splits squality consists of 100 stories that are split 39/25/36 across the train/validation/test splits (or, equivalently, 195/125/180 document-question pairs). we assign stories to splits to be consistent with the quality dataset @xcite , so stories that appear in both datasets are assigned to the same split. squality contains a similar number of summaries to @xcite , another crowdsourced summarization dataset, but squal-ity contains four references per example and thus fewer input documents. this difference in allocation arises from the crowdsourcing protocol: in creating squality, we have writers review each other's work while in creating qmsum, the authors manually review all responses. protocols wherein workers review each other work are more scalable. having multiple references per input is useful for model evaluation, as automatic metrics such as rouge were originally developed on multireference datasets. while naive multi-reference rouge still correlates poorly with human judgments of quality for squality (see section 6), having a diverse set of references opens up opportunities for the development of new evaluation metrics that take into account the diversity of acceptable summaries for a given input, even in the question-focused setting. length documents are an average of 5200 tokens long (std. 522) without punctuation, 9 with a range from 3473 to 6165-similar to the chapters version of booksum, and shorter than qmsum. responses average 237 tokens long (std. 133), corresponding to a compression ratio of 95.4%. the plot summaries have an average length of 442 tokens and are comparable in length to those of booksum. the other responses are shorter with an average length of 186 tokens, but are still longer than the summaries in qmsum. ## human evaluation automatic metrics for evaluating text summarization have been well-documented as correlating poorly with human judgments of various quality @xcite @xcite . as such, we accompany automatic evaluation of the baseline systems with human evaluation. we ask workers to rate the quality of outputs from bart and bart+dpr on the test data. for each task, we show the worker a story and for each of its ve questions, two model-generated summaries and a human reference. workers rate each summary for three properties: correctness, coverage, and overall quality. each property is rated on a scale from 1-100, similar to direct assessment ratings in machine translation @xcite . workers are instructed to assign ratings that align with their preference rankings between systems (sakaguchi and van durme, 2018). we annotate 20 stories (100 questions) with three upwork workers per story. finally, we average property ratings across annotators. the worker details and property denitions are available in appendix e. we present results of the human evaluation in ## conclusion we present squality, a long-input dataset for abstractive question-focused summarization. because the summaries are crowdsourced rather than found, we can use input documents that are of an accessible domain and under an open license to avoid common issues with existing summarization datasets. our crowdsourcing protocol gives multiple summaries and references per input while making the cost of data collection more tractable. the squality dataset is available at https: //github.com/nyu-mll/squality . baseline results with competitive public medium-scale pretrained models suggest that the dataset remains beyond the capabilities of such systems. our best performing model is an extract-then-summarize model where we use the questions to retrieve story sentences as input. the performance of proprietary larger-scale models remains an open question, and may depend significantly on whether such models can process the full stories without truncation. developing more robust architectures for processing long documents than longformer might lead to straightforward improvements on squality. creating efcient and effective methods for evaluating summaries of long input documents remains an open issue. given the poor correlation of existing automatic metrics with human judgments of model outputs, we expect that these automatic metrics will provide a very weak signal for progress on squality. we recommend that researchers using squality evaluate their summarization systems by having human annotators read a selection of our source stories and compare model outputs on those stories. to facilitate this, we will make our templates for human evaluation available, as well as the judgments from our human evaluation experiments to develop better automatic evaluation metrics.