M ovie S um: An Abstractive Summarization Dataset for Movie Screenplays (2024)

Abstract: Movie screenplay summarization is challenging, as it requires an understanding of long input contexts and various elements unique to movies. Large language models have shown significant advancements in document summarization, but they often struggle with processing long input contexts. Furthermore, while television transcripts have received attention in recent studies, movie screenplay summarization remains underexplored. To stimulate research in this area, we present a new dataset, MovieSum, for abstractive summarization of movie screenplays. This dataset comprises 2200 movie screenplays accompanied by their Wikipedia plot summaries. We manually formatted the movie screenplays to represent their structural elements. Compared to existing datasets, MovieSum possesses several distinctive features: 1) It includes movie screenplays which are longer than scripts of TV episodes. 2) It is twice the size of previous movie screenplay datasets. 3) It provides metadata with IMDb IDs to facilitate access to additional external knowledge. We also show the results of recently released large language models applied to summarization on our dataset to provide a detailed baseline.

## introduction large language models have shown significant improvements in abstractive summarization in recent years @xcite @xcite , aiming to produce a concise and coherent summary of the input document. however, these models often struggle when the input context is long, particularly when the relevant information is distributed across the document @xcite . to better understand this phenomenon and to advance research, datasets are needed that not only contain long-form documents but also have the property that important information is dispersed throughout the document. movie screenplays have these characteristics: to generate a faithful summary, an understanding of characters and events across the entire length of the screenplay is required. more recently, narrative summarization research has focused on tv shows and books @xcite , with less attention given to movie screenplays @xcite . notably, @xcite introduced a dataset of tv show transcripts which has gained considerable interest and was included in a long document summarization benchmark @xcite . but unlike movie screenplays, tv episode transcripts tend to be relatively short and predominantly comprise spoken dialogue with minimal scene or character descriptions. additionally, they are not selfcontained, as the events or characters from previous episodes can be referred to. in contrast, movie screenplays are structured documents with various screenplay elements such as scene headings, locations, character names, dialogues and detailed scene descriptions. these are written by screenwriters and are characteristically formatted to denote each element. the largest current movie screenplay dataset (gorinski and lapata, 2015, 2018) comprises 917 automatically formatted screenplays (scriptbase-j), with the most recent movie from 2013. we built moviesum, a new movie screenplay dataset for abstractive summarization, which consists of 2200 movies, more than twice the size scriptbase-j. importantly, our new dataset has been formatted using a professional script writing tool and paired with wikipedia plot summaries. each movie is also tagged with its imdb ids to facilitate the collection of other external knowledge in rhw future. the dataset consists of movies spanning a wide range of genres from 1930 to 2023. we provide a detailed description of moviesum, including the steps for collecting and filtering screenplays and statistics and comparison with other narrative datasets. we conduct extensive experiments to evaluate the performance of stateof-the-art summarization models on moviesum, demonstrating its utility as a benchmark dataset for narrative summarization research. experiments suggest that recent models struggle with long abstractive summarization, and we hope that our efforts will inspire further research in this area. furthermore, we provide qualitative analyses of how the structure of the screenplay can be utilized in generating summaries. ## the moviesum dataset we present moviesum, a movie screenplay abstractive summarization dataset that consists of 2200 movie screenplay-summary pairs. all movie screenplays in the dataset are in english. ## dataset analysis this results in a dataset consisting of 2200 manually formatted movie screenplays along with their corresponding summaries. the average length of the screenplays is 29k words, with an average summary length of 717 words. importantly, this dataset is twice the size of the previously available movie screenplays dataset with formatted movie screenplays (gorinski and lapata, 2015). figure 1a . illustrates the genre distribution of the movies within the dataset and showcases the broad range of genres. in figure 1b , the distribution of release years is depicted, revealing that the movies span a wide range of years, with a substantial number of them originating in recent years. to study the abstractiveness of the summary, we report the percentage of novel n-gram in ## experiments we evaluate the moviesum dataset using several baselines and state-of-the-art neural abstractive summarization models. we first report the lead-n baseline, which simply outputs the first n tokens of the movie script as the movie summary. we varied the value n to understand the impact of summary length on performance and report results for lead-512, lead-768, and lead-1024. for the extractive baseline, we used textrank @xcite , a graph-based unsupervised extractive summarization method. for instructiontuned large language models, we used vicuna 1.5 13b 16k @xcite , built on llama-2 @xcite , and flan-ul2 @xcite in a zero-shot setting. for fine-tuned models with long inputs, we utilized longt5 @xcite , pegasus-x @xcite , and the longformer encoder-decoder (led) model @xcite . we fully finetuned these models and report results on the test set. the implementation details of the models are mentioned in appendix a. ## discussion and conclusion we introduce the moviesum dataset, comprising formatted and recent movie screenplays paired with wikipedia summaries. our experiments demonstrate that it is a challenging dataset even for a large language model with a long input length. we hope that moviesum will enable future research in the area of movie screenplay understanding and abstractive summarization.