N arra S um: A Large-Scale Dataset for Abstractive Narrative Summarization (2022)

Abstract: Narrative summarization aims to produce a distilled version of a narrative to describe its most salient events and characters. Writing a summary for a narrative is challenging as it requires an understanding of event causality and character behaviors. To encourage research in this direction, we propose NarraSum, a large-scale narrative summarization dataset. It contains 122K narratives, which are collected from the synopses of movies and TV episodes with diverse genres, and their corresponding abstractive summaries. Experiments show that there is a large performance gap between humans and the state-of-the-art summarization models on NarraSum. We hope that this dataset will promote future research in summarization, as well as broader studies of natural language understanding and generation. The dataset is available at https://github.com/zhaochaocs/narrasum.

## introduction a narrative is a story (e.g., a novel or a movie) composed of events and characters @xcite . narrative summarization aims to produce a distilled version of a narrative, either extractively or abstractively, to contain its most salient events and major characters @xcite . this ability is especially crucial for the understanding of narratives, and in general, the understanding of human behaviors and beliefs @xcite . practically, a summary of a narrative can enable a reader to quickly discern the key points, which is useful in real-world scenarios such as content recommendations and advertisements. while text summarization has been explored for over decades, most existing studies focus on summarizing news @xcite @xcite or structured documents (e.g., scientific papers @xcite ). these documents have specific writing styles. for document: ( https://bigbangtheory.fandom.com/wiki/the_big_bran_hypothesis ) setting their dinner of thai food, sheldon gives the group a lecture of the use of the fork in thai history. a little later, penny talks with leonard in the hallway about her work at the cheesecake factory. she then asks leonard to sign for a piece of furniture while she is out. @xcite it turns out the furniture is bigger than they had expected. the delivery man does not help them, so leonard and sheldon are forced to carry it up the stairs the themselves since the elevator doesn't work. sheldon's only idea involves using a green lantern power ring. finally, they eventually succeed in getting it up the stairs to her apartment. while there, sheldon sees that penny's apartment is a complete mess and insists on tidying up. @xcite leonard get up the next morning and sheldon tells him that he slept well. leonard remarks that a well known folk cure for insomnia is to break into your neighbor's apartment and clean. sheldon asks if that was sarcasm. penny awakens to find out that her apartment in a well ordered state and screams about those geeky bastards. penny charges into sheldon and leonard's apartment in a fit of rage about them coming into her place while she was sleeping. she demands her key back. @xcite later, penny runs into raj in the hallway and talks to him about being upset over what happened (although he doesn't reply as he has selective mutism). penny decides to forgive them while raj was thinking; "boy, her hair smells nice" and "maybe my mother was right. maybe i should marry an indian girl. we would have the same cultural background and she could sing the same lullabies my mother sang to me". penny then hugs raj, much to his surprise. @xcite summary: ( https://en.wikipedia.org/wiki/the_big_bang_theory_(season_1)#ep2 ) when sheldon and leonard drop off a box of flat pack furniture that came for penny, sheldon is deeply disturbed at how messy and disorganized her apartment is. later that night, while penny sleeps, the obsessive-compulsive sheldon, unable to sleep, sneaks into her apartment to organize and clean it. leonard finds out and reluctantly helps him. the next morning, penny is furious to discover they had been in her apartment. sheldon tries to apologize to penny but fails by remarking that leonard is a "gentle and thorough lover". later, penny encounters raj in the hallway. though he cannot talk to penny, she calms down whilst telling him about the issue, reasoning the guys were just trying to help her, and hugs raj. then leonard apologizes, prompting penny to forgive and hug him. figure 1 : example of the narrative summarization task. the input is a narrative text (denoted by "document", pictures are not included), and the output is a summary containing its salient events and characters. understanding an entire narrative faces unique challenges. a narrative organizes the story into a sequence of events (i.e., plot) in a chronological and causal order @xcite . events unfold due to the actions of characters and other event participants, or external forces in stories @xcite . to identify the salient events, a model needs to understand both plot and characters. from the plot's perspective, the model needs to understand the causal and temporal relationships between events, as well as how the plot develops from the beginning to the end @xcite . from the character's perspective, the model needs to understand the characters' profiles (e.g., personalities, roles, and interpersonal relationships), and how their desires and actions drive the story forward. figure 1 illustrates the importance of understanding the entire narrative for summarization. in this example, the main event is "sheldon cleans penny's apartment and gets leonard in trouble", which is included in the summary. the side event "penny speaks to raj and forgives leonard" is also included since it is the consequence and ending of the main event. whereas, "sheldon gives a lecture of fork" is not included as it does not impact the development of the plot. besides the main events, the summary also explains sheldon's motivation to clean the apartment. a large-scale high-quality dataset is essential to promote research on this topic. unfortunately, different from other domains, such as news and scientific papers, where the document and summary can be found from the same data source, narrative documents and their corresponding summaries are usually spread in separate sources. previous studies collect document-summary pairs of narrative by either creating summaries manually @xcite or matching titles between documents and summaries followed by a manual inspection @xcite , making it challenging to enlarge the resulting datasets. in this work we propose an automatic data construction framework to build a narrative summarization dataset with both large scale and high quality. specifically, we first collect narratives from plot descriptions of movies or tv episodes through online resources. we choose the plot description because it describes the overall narrative of the movie or tv episode, including the story arcs and major characters. this source is also widely used in narrative-related studies @xcite @xcite . after data collection, we build an align-and-verify pipeline to automatically align plot descriptions of the same movie or tv episodes from different sources. finally, we construct document-summary pairs by treating the long plot description as the document to be summarized and the shorter one (of the same movie or tv episode) as the corresponding summary. after filtering out low-quality document-summary pairs, we build narrasum, a large-scale dataset that contains around 122k narrative documentsummary pairs in english. our data construction framework is generic and thus can potentially be applied to other languages as well. to gauge the feasibility of narrasum for the narrative summarization task, we explore different characteristics of this dataset. we observe that compared with other summarization datasets, the narratives in narrasum are of diverse genres, and the summaries are more abstractive and of varying lengths. furthermore, rather than focusing on a particular part of the document (as in other summarization datasets), the summaries in narrasum are designed to cover the entire narratives. it brings new challenges to current summarization methods. we investigate the performance of several strong baselines and state-of-the-art summarization models on narrasum. results show that there is a large gap between human and machine performance in various dimensions, demonstrating that narrative summarization is a challenging task. the contributions of this paper are four-fold: ## data construction we propose an automatic data construction framework to create a narrative summarization dataset. to this end, we first collect plot descriptions of movies and tv episodes from multiple resources as narratives (section 2.1). we then align plot descriptions in these resources that refer to the same movie or tv episode (section 2.2). finally, we filter the aligned data to construct high-quality documentsummary pairs. (section 2.3). we describe the details of each step as follows. ## data analysis this section provides basic statistics of narra-sum. we then analyze the dataset in terms of the distribution of salient information and abstractiveness of summaries. finally, we conduct a human assessment to evaluate the quality of narrasum. ## baseline models we investigate the performance of several baselines and state-of-the-art neural summarization models on narrasum. we include both extractive and abstractive models. for extractive models, we use the following methods: random selects n sentences from the document randomly. lead selects the top-n sentences from the document to compose the summary. this is a strong baseline for news summarization. textrank @xcite ) is a graphbased extractive summarization model based on pagerank @xcite in a graph representation of sentences. lexrank @xcite ) is another graph-based extractive summarization model based on eigenvector centrality . hsg @xcite ) is a heterogeneous graphbased neural extractive summarization model that uses word co-occurrence to enhance sentence contextual representation. presumm (liu and lapata, 2019) relies on a pretrained language model to enhance the sentence representation during text encoding and extractive summarization. we choose bert @xcite , roberta @xcite , and long-former @xcite as the pre-trained models. bert and roberta limit the input length to be shorter than 512 tokens, while longformer can accept up to 4,096 tokens. for abstractive models, we use the following pre-trained sequence-to-sequence models: bart @xcite , t5 @xcite , pegasus @xcite , and led @xcite . the input length of the first three models is limited to 512 (base version) or 1,024 (large version). led uses longformer as the encoder and therefore can accept up to 4,096 tokens as input. ## analysis we perform a series of analyses about the summary position and character consistency. for a fair comparison among models, we only choose test instances where the length of the document is shorter than the maximum input length of these models (1,024 tokens). ## application to other tasks besides presenting narrasum as a benchmark for narrative summarization, we further explore the broader benefits of this dataset to narrative-related tasks. we first investigate whether pre-training on narrasum can improve performance on other narrative summarization tasks. to this end, we first pre-train a bart-large model on narrasum and then finetune it on novel chapter and booksum-paragraph. we compare with the finetuned models without pre-training on narrasum. as shown in we then investigate if narrasum can help the model learn general knowledge of narrative understanding and summarization. for this, we first pre-train a bart-large model on narrasum and then apply it to several downstream tasks in a zero-shot manner. we choose five tasks that are designed for narrative understanding, i.e., mctest @xcite , movieqa @xcite , liscu @xcite , cbt @xcite , and quail @xcite , and one task for narrative summarization, i.e., reddit tifu @xcite . for each task, we provide the corresponding task description, method, and evaluation measure in appendix a.3. we use models trained on the summarization task to solve these tasks in a zero-shot manner. in other words, we do not use any training data from these tasks. for discriminative tasks, we first convert the (question, answer) pair into a statement using a t5 model @xcite , and then evaluate the probability of generating each statement conditioned on the document @xcite . we choose the candidate with the highest generation probability as the predicted answer. models are evaluated using accuracy. for the summarization task, we directly apply the trained model to create the summary. models are evaluated using the rouge-1 f measure. we compare the model pre-trained on nar-rasum with those pre-trained on other narrative summarization datasets such as novel chapter and booksum. as shown in ## conclusion we present narrasum, a large-scale narrative summarization dataset that contains plot descriptions of movies and tv episodes and the corresponding summaries. narratives in narrasum are of diverse genres, and the summaries are highly abstractive and of varying lengths. summarizing the narratives in narrasum requires narrativelevel understanding, which poses new challenges to current summarization methods. experiments show that current models struggle with creating high-quality narrative summaries. we hope that narrasum will promote future research in text summarization, as well as broader nlp studies such as machine reading comprehension, narrative understanding, and creative writing.