S umm S creen: A Dataset for Abstractive Screenplay Summarization (2022)

Abstract: We introduce SummScreen, a summarization dataset comprised of pairs of TV series transcripts and human written recaps. The dataset provides a challenging testbed for abstractive summarization for several reasons. Plot details are often expressed indirectly in character dialogues and may be scattered across the entirety of the transcript. These details must be found and integrated to form the succinct plot descriptions in the recaps. Also, TV scripts contain content that does not directly pertain to the central plot but rather serves to develop characters or provide comic relief. This information is rarely contained in recaps. Since characters are fundamental to TV series, we also propose two entity-centric evaluation metrics. Empirically, we characterize the dataset by evaluating several methods, including neural models and those based on nearest neighbors. An oracle extractive approach outperforms all benchmarked models according to automatic metrics, showing that the neural models are unable to fully exploit the input transcripts. Human evaluation and qualitative analysis reveal that our non-oracle models are competitive with their oracle counterparts in terms of generating faithful plot events and can benefit from better content selectors. Both oracle and non-oracle models generate unfaithful facts, suggesting future research directions.

## introduction abstractive summarization aims to produce a summary that concisely expresses key points of the input document rather than merely extracting pieces of it. existing datasets are constructed from various domains, such as news @xcite transcript: recap: figure 1: excerpts from an example from summ-screen. the transcript and recap are from the tv show "the big bang theory". generating this sentence in the recap requires discerning the characters' feelings (clues in the transcript are underlined) about playing the board game (references are shown in red). colored boxes indicate utterances belonging to the same conversations. @xcite @xcite , online forums @xcite , meeting dialogues @xcite , and webpages @xcite . however, few datasets exist for abstractive summarization of narrative text, which focuses on entities and dialogue among entities, with plot details often communicated indirectly via dialogue. in this work, we build summscreen, an abstractive summarization dataset combining tv series transcripts and episode recaps. figure 1 shows an example from summscreen. several aspects of summscreen make it a challenging testbed for abstractive summarization. first, the relationship between character dialogue and plot details is not straightforward. plot events are often expressed indirectly in dialogue, and dialogue contains other information that is not directly relevant to the plot, such as character development and humor. also, a typical episode has multiple subplots that proceed in parallel, with consecutive scenes often describing different subplots. solving summscreen requires drawing information from utterances across a wide range of the input and integrating the information to form concise plot descriptions. moreover, since actual tv episodes ground their scripts with audio-visual accompaniment, many details may be omitted from the transcript itself. this omission of details and the other challenging aspects mentioned above have inspired research into other nlp tasks on tv show transcripts, such as entity tracking @xcite ) and coreference resolution @xcite . another prominent characteristic of tv series transcripts is their focus on characters. to reflect this aspect, we propose two entity-centric metrics to evaluate the quality of generated plot summaries. one is based on bags of characters, which measures the overlap of the characters that appear in both the generated and reference recaps. the other metric measures character relations: the overlap of cooccurrences of character pairs in generations and recaps. we empirically evaluate several types of methods on summscreen. we consider nearest neighbor models, which look up similar transcripts or recaps, neural abstractive summarization models, and hybrid models, which use the nearest neighbor models as content selectors followed by abstractive summarization. oracle extractive approaches outperform all models on all the automatic metrics. these results suggest that the benchmarked methods are unable to fully exploit the input transcripts and that improving content selection may be a promising research direction. human evaluations show that our non-oracle hybrid models are competitive with their oracle counterparts in terms of generating faithful plot events. hybrid models may be promising approaches for future research. qualitative analysis shows that neural models tend to generate generic summaries, hybrid models can benefit from better content selection, and hybrid models sometimes generate unfaithful details. ## related work there has been prior work on extractive screenplay summarization @xcite , and analyzing crime drama @xcite . the majority of tv show transcripts are dialogues, relating our work to prior work on dialogue and meeting summarization. relevant datasets have been studied for medical dialogues @xcite , chitchat (samsum; @xcite , podcasts @xcite , meetings (ami; @xcite @xcite @xcite , livestreams (streamhover; @xcite , online forums (forumsum; @xcite and news interviews (mediasum; @xcite . there have been attempts in summarizing longform text (other than screenplays), such as books @xcite , scientific articles (pubmed and arxiv; @xcite , multiple news articles (multi-news; @xcite , opinionated text (rottentomatoes; @xcite , patents @xcite , tv show stories (tvrecap; chen and gimpel, 2021) and (extractive summarization of) chapters of novels @xcite . more detailed discussion on the differences between these datasets and summ-screen is in the next section. recently there have been efforts on adapting resources for tv shows for different tasks, including question answering @xcite , speaker identification @xcite , sarcasm detection @xcite , emotion detection @xcite , character relation extraction @xcite , and story generation @xcite . ## summscreen an instance in summscreen contains a transcript from tv series and its corresponding recap. the transcripts consist of dialogue utterances with speaker names, and descriptions of scenes or character actions. the recaps are human-written summaries of the corresponding transcripts. figure 1 shows an example in summscreen from the tv show "the big bang theory". the transcript documents a dialogue involving four characters (sheldon, leonard, penny, and amy) about playing a board game, and the recap summarizes the dialogue into sentences. ## approaches in this section, we describe modeling approaches that we benchmark on summscreen. we note that since the meaning of sentences in transcripts is highly context-dependent, extractive summarization approaches are not expected to be useful for this dataset. we report the results from nearest neighbor-based extractive summarizers mostly for characterizing the dataset. ## analysis 7 effect of combining fd and tms we study the effect of transfer learning using these two resources. when doing so, we use the training and development sets constructed from both resources, and at test time, we evaluate models on the official test splits. we experiment with the oracle hybrid model and report results in ## conclusion we construct summscreen, which contains pairs of tv show transcripts and recaps. we qualitatively analyze the challenging aspects of our dataset. we propose two entity-centric metrics to evaluate generated summaries with one focusing on character overlap and the other focusing on overlap of charac-ter pairs. empirically, we benchmark several neural models and nearest neighbor models for characterizing our datasets, finding that an oracle extractive summarizer gives the strongest performance according to the automatic metrics. human evaluations show that the non-oracle hybrid model is competitive at generating faithful topics. qualitative analysis shows that the hybrid model can benefit from better content selectors and both oracle and non-oracle hybrid models suffer from generating inaccurate details, highlighting several directions for future research.