Select and Summarize: Scene Saliency for Movie Script Summarization (2024)

Abstract: Abstractive summarization for long-form narrative texts such as movie scripts is challenging due to the computational and memory constraints of current language models. A movie script typically comprises a large number of scenes; however, only a fraction of these scenes are salient, i.e., important for understanding the overall narrative. The salience of a scene can be operationalized by considering it as salient if it is mentioned in the summary. Automatically identifying salient scenes is difficult due to the lack of suitable datasets. In this work, we introduce a scene saliency dataset that consists of human-annotated salient scenes for 100 movies. We propose a two-stage abstractive summarization approach which first identifies the salient scenes in script and then generates a summary using only those scenes. Using QA-based evaluation, we show that our model outperforms previous state-of-the-art summarization methods and reflects the information content of a movie more accurately than a model that takes the whole movie script as input.

## introduction abstractive summarization is the process of reducing an information source to its most important content by generating a coherent summary. previous work has primarily focused on news @xcite , meetings @xcite , and dialogues @xcite , but there is limited prior work on summarizing long-form narrative texts such as movie scripts @xcite . long-form narrative summarization poses challenges to large language models @xcite @xcite both in terms of memory complexity and in terms of 1 our dataset and code is released at https://github.com/saxenarohit/select_summ . several methods have previously relied on content selection for summarization to reduce the input size by either performing content selection implicitly using neural network attention @xcite @xcite or explicitly @xcite @xcite by aligning the source document with the summary using metrics such as rouge @xcite . unlike for news articles, the implicit attention-based method is problematic for movie scripts, as current methods cannot reliably process text of such length. on the other hand, current explicit methods are neither optimized nor evaluated for content selection using gold-standard labels. in addition, considering the large number of sentences in movies that contain repeated mentions of characters and locations, a method based on a lexical overlap metric such as rouge creates many false positives. crucially, all these methods use source-summary alignment as an auxiliary task without actually optimizing or evaluating this task. for news summarization, @xcite created crowd-sourced development and test sets for the evaluation of proposition-level alignment. however, news texts differ from movie scripts both in length and in terms of the rigid inverted pyramid structure that is typical for news articles. for movie scripts, @xcite proposed a specialized alignment method which they evaluated on a set of 10 movies. however, they do not perform movie script summarization. movie scripts are structured in terms of scenes, where each scene describes a distinct plot element and hapening at a fixed place and time, and involving a fixed set of characters. it therefore makes sense to formalize movie summarization as the identification of the most salient scenes from a movie, followed by the generation of an abstractive summary of those scenes @xcite . hence we define movie scene saliency based on whether the scene is mentioned in the summary i.e., if the scene is mentioned in the summary, it is considered salient. using scene saliency for summarization is therefore a method of explicit content selection. in this paper, we first introduce mensa, a movie scene saliency dataset that includes human annotation of salient scenes in movie scripts. our annotators manually align wikipedia summary sentences with movie scenes for 100 movies. we use these gold-standard annotations to evaluate existing explicit alignment methods. we then propose a supervised scene saliency classification model to identify salient scenes given a movie script. specifically, we use the alignment method that performs best on the gold-standard data to generate silverstandard labels on a larger dataset, on which we then train a sequence classification model using scene embeddings to identify salient scenes. we then fine-tune a pre-trained language model using only the salient scenes to generate movie summaries. this model achieves new state-of-the-art summarization results as measured by rouge and bertscore @xcite . in addition to that, we evaluate the generated summaries using a question-answer-based metric @xcite and show that summaries generated using only the salient scenes outperform those generated using the entire movie script or baseline models. ## mensa: movie scene saliency dataset we define the saliency of a movie scene based on the mention of the scene in a user-written summary of the movie. if the scene appears in the summary, then it is considered salient for understanding the narrative of the movie. by aligning summary sen-tences to movie scenes, we identify salient scenes and later use them for movie summarization. the mensa dataset consists of the scripts of 100 movies and respective wikipedia plot summaries annotated with gold-standard sentence-toscene alignment. we selected 80 movies randomly from scriptbase (gorinski and lapata, 2015) and added 20 recently released, manually corrected movie scripts, which all had wikipedia summaries. both mensa and scriptbase datasets are movie scripts datasets and differ from other dialogue/narrative datasets such as summscreenfd @xcite , the foreverdreaming subset of the summscreen dataset as used in the scrolls benchmark @xcite . summscreenfd is dataset of tv show episodes and consists of crowd-sourced transcripts and recaps. in contrast, the movie scripts in our dataset were written by screenwriters and the summaries were curated by wikipedia. it is important to note that movies and tv shows have different storytelling structures, number of acts, and length. summscreenfd has shorter input texts and summaries compared to movie scripts as shown in ## scene saliency classification model using the set of generated silver-standard labels for scene saliency, we train a neural networkbased classification model to predict scene saliency. we formulate this task as a sequence labeling task where the model takes a sequence of scenes transformer encoder transformer decoder scene 1 scene 2 scene n-1 scene encoder scene encoder scene encoder transformer layers salient scene classifier scene n scene encoder scene 1 scene 2 scene k movie summary positional embeddings scene saliency model movie script 1 0 0 1 salient scenes summarization model figure 1 : the architecture of the scene saliency detection and summarization models. the models are trained in a pipeline where salient scene detection is trained separately. @xmath0, s 2 , ..., s n } as input and predicts a sequence of binary labels @xmath1, y 2 , ..., y n } denoting whether a scene is salient. the model consists of two components, as shown in figure 1 . the first component is a scene encoder which computes scene representations by concatenating the sentences in the scene and encodes them using a pre-trained language model. next, to learn contextual scene representation across the whole movie, we further encode the scene embeddings generated by the scene encoder using a transformer @xcite block (l layers stacked), with unmasked self-attention initialized with random weights @xcite . to preserve the sequence of the scenes, we add positional encodings to scene representations obtained from the first component. the final contextualized representation of the scenes is then used to classify whether scenes are salient or not. the model is trained for binary sequence labeling using the binary cross-entropy loss. ## summarization using salient scenes we now investigate the benefit of using only salient scenes for the abstractive summarization of movie scripts. we formulate this task as a sequence-to-sequence generation problem. formally, given a movie with a set of salient scenes @xmath2, s 2 , ..., s k }, the goal is to generate a target summary @xmath3, s 2 , ..., s m }. as the input length of the salient scenes is still quite large as shown in figure 2 , we use a longformer encoder-decoder (led) architecture @xcite . to handle long input sequences, led uses efficient local attention with global attention for the encoder. the decoder then uses the full self-attention to the encoded tokens and to previously decoded locations to generate the summary. ## automatic qa-based evaluation metrics like rouge (lexically based) and bertscore (embedding based) are good for comparing the topic similarity between the reference and generated summaries, but fail to compare content-based factual consistency. to further evaluate the performance of our model, we used qaeval @xcite , a question-answering-based evaluation that generates question-answer pairs using the reference summaries. it then uses the modelgenerated summaries (candidates) to answer these questions, thereby measuring information overlap. it reports two standard answer verification methods used by squad, f1 and exact match (em) @xcite , averaged over all questions for all model-generated summaries. before the final evaluation, we filtered the generated questions using a question filtering method similar to @xcite , which is useful for removing spurious questions/answers (for example answers consisting of personal pronouns and wh-pronouns). ## zero-shot on summscreen-fd we further investigate the performance of the scene saliency classifier on summscreenfd @xcite as used in scrolls benchmark @xcite . summscreenfd consists of transcripts of tv show episodes with human-written recaps. we performed a zero-shot classification of the salient scenes on the summscreenfd and used only salient scenes to fine-tune led for summarization. we compare the results with state-of-the-art methods on the dataset and report rouge scores. we observe that our model achieves comparable results to the state of the art on the summscreenfd dataset as shown in ## discussion and conclusion in this paper, we introduced a dataset of 100 movies in which movie plot summaries are manually aligned with scenes in the corresponding movie script. our dataset can be used to evaluate content selection strategies and extractive summarization for movie scripts. using this dataset, we proposed a scene saliency classification model for the automatic identification of salient scenes in a movie script and introduced an abstractive summarization model that only uses the salient scenes to generate the movie summary. our experiments showed that the proposed model achieves a significant improvement over the previous state of the art on the scriptbase corpus for movie script summarization and performs comparable to the state of the art on the summscreenfd dataset using zero-shot salient scene detection. our work demonstrates that the output of a summarization model can improve when content selection is performed (by using only the salient scenes). a good content selection strategy can in principle reduce the input size without compromising the quality of the generated output. as a result of the smaller input size, the computational and memory requirements of the underlying large language model can be significantly reduced.