BOOKSUM : A Collection of Datasets for Long-form Narrative Summarization (2022)

Abstract: The majority of existing text summarization datasets include short-form source documents that lack long-range causal and temporal dependencies, and often contain strong layout and stylistic biases. While relevant, such datasets will offer limited challenges for future text summarization systems. We address these issues by introducing BOOKSUM, a collection of datasets for long-form narrative summarization. Our dataset covers documents from the literature domain, such as novels, plays and stories, and includes highly abstractive, human written summaries on three levels of granularity of increasing difficulty: paragraph-, chapter-, and book-level. The domain and structure of our dataset poses a unique set of challenges for summarization systems, which include: processing very long documents, non-trivial causal and temporal dependencies, and rich discourse structures. To facilitate future work, we trained and evaluated multiple extractive and abstractive summarization models as baselines for our dataset.

## introduction text summarization aims at condensing long documents into a short, human-readable form which contains only the salient parts of the source. leveraging the cutting-edge findings in natural language processing, such as multi-task learning methods @xcite , pre-training strategies @xcite , and memory-efficient architectures @xcite ), text summarization has seen substantial progress. the majority of papers published in the field focus on summarizing newswire documents from popular datasets, such as cnn/dailymail @xcite . other domains gaining interest of the research community are scientific and legal documents, with notable datasets being arxiv/pubmed @xcite and big-patent @xcite . while the performance of state-of-the-art methods on those datasets is impressive, the mentioned domains have inherent shortcomings and thus pose limited challenges for future generations of text summarization systems. first, the length of summarized documents is limited, ranging from a few hundred words in case of news articles, to a few pages for scientific documents and patent applications. in most cases, such short-form documents can be quickly read by humans, thus limiting the practical value of automatic summarization systems. second, the domains under consideration impose strict requirements regarding the document's layout and stylistic features foot_0 . statements follow a logical order and all facts are offered explicitly, leaving limited space for interpretation and reasoning. additionally, such constraints, can introduce layout biases into the datasets which later dominate the training signal of the summarization systems. the leadbias present in news articles being one example of such effects @xcite . third, documents in the mentioned domains lack long-range causal and temporal dependencies, and rich discourse structures. due to the limited length and fact-centric style of writing, most causal dependencies span only a few paragraphs, temporal dependencies are organized in a monotonic fashion where newly introduced facts refer only to previously stated information, and document lacks features such as parallel plot lines. in this work we address the shortcomings of existing datasets and introduce booksum, a collection of data resources for long-form narrative summarization @xcite . the data covers documents from the literature domain, including stories, plays, and novels (fig. 1 ), each provided with a highly abstractive, human-written summary. leveraging the characteristics of fiction writing, booksum introduces a set of new challenges for summarization systems: processing longform texts ranging up to hundreds of pages, understanding non-trivial causal and temporal dependencies spread out through the entirety of the source, handling documents with rich discourse structure which include parallel plots or changes between narration and dialogue, and generating highly abstractive and compressive summaries. solving such challenges will require progress in both automatic document understanding and processing of long inputs. to support incremental progress, the booksum collection includes examples on three levels of granularity with increasing difficulty: 1) paragraph-level, with inputs consisting of hundreds of words and short, single-sentence summaries, 2) chapter-level, with inputs covering several pages and multi-sentence summaries, 3) book-level, with inputs spanning up to hundreds of pages and multiparagraph summaries. the hierarchical structure of the dataset, with aligned paragraph, chapter, and book-level data, makes it a viable target for singleand multi-document summarization approaches. to demonstrate the new set of challenges for text summarization models introduced by book-sum and lay the groundwork for future research, we evaluated several state-of-the-art extractive and abstractive summarization architectures on the newly introduced task. we share the data preparation scripts here: https://github.com/ salesforce/booksum . ## related work the availability of digital documentation has translated into a number of novel, large-scale datasets for text summarization that span a variety of domains. in the news domain, @xcite introduced a corpus of news articles from the new york times magazine with summaries written by library scientists. @xcite collected articles from the cnn and dailymail portals with multisentence article highlights repurposed as reference summaries. @xcite aggregated articles from the bbc website with highly abstractive, single sentence summaries. @xcite introduced a dataset spanning 38 news portals, with summaries extracted from the websites metadata. in the academic article domain, @xcite collected scientific articles from the arxiv and pub-meb article repositories and used paper abstracts as reference summaries. wang et al. (2020) aggregated a set of articles in the medical domain related to the covid-19 pandemic, also using paper abstracts as reference summaries. @xcite introduced a multi-domain collection of scientific articles each with two associated summaries, one covering the article's contributions, the other explaining the context of the work. related to dialogue summarization, @xcite repurposed image captioning and visual dialogue datasets to create a summarization dataset containing conversations describing an image, with image captions considered the summaries. @xcite introduced a corpus of conversations between hired annotators designed to mimic interactions on a messaging application with human written summaries. in the legal domain, @xcite has collected a. large collection of patent filings with associated, author-written invention descriptions. despite the increased interest in the broader field of text summarization, little work has been done in summarizing stories and novels. in @xcite , the authors focused on generating extractive overviews of short works of fiction. the work proposed two modeling approaches, one utilizing decision trees the other based on a manually designed system of rules with experiments conducted on a set of 23 short stories. @xcite introduced the task of book summarization along with a set of resources and baselines. the authors collected and curated a set of 50 books from the gutenberg project with two human-written summaries associated with each book collected from online study guides. more recently, @xcite tackled the problem of generating character descriptions based on short fiction stories. the authors collected a dataset of stories with associated, author-written summaries from online storysharing platforms and proposed two baseline methods for solving the task. ladhak et al. (2020) ex-plored the problem of content selection in novel chapter summarization. the authors studied different approaches to aligning paragraphs from book chapters with sentences from associated summaries and created a silver-standard dataset for extractive summarization. the work also studied the performance of extractive models on the task. our work extends the efforts made by @xcite . the booksum corpus prioritizes abstractive summarization and offers aligned data on three levels of granularity (paragraph, chapter, full-book), substantially increasing the number of available examples. we also benchmark the performance of state-of-the-art extractive and abstractive methods on all introduced data subsets. ## dataset in this section we describe the data sources and pre-processing steps taken to create the book-sum data collection and conduct an in-depth analysis of the collected resources. ## experiments to motivate the challenges posed by the book-sum corpus, we study the performance of multiple baseline models, both extractive and abstractive, on the different levels of alignment: paragraph, chapter and books. we refer to these levels of alignment as booksum-paragraph, booksum-chapter, and booksum-book accordingly. ## conclusions in this work we introduced booksum, a collection of datasets for long-form narrative summarization. booksum includes annotations on three levels of granularity of increasing difficulty: paragraph, chapter, and full-book. through a quantitative analysis we compare our dataset to existing summarization corpora and show that booksum sets new challenges for summarization methods. we trained extractive and abstractive baseline models leveraging state-of-the-art pre-trained architectures to test the performance of current methods on the task of long-narrative summarization and to enable easy comparison with future methods. we hope our dataset will contribute to the progress made in the field of automatic text summarization.