A Modular Approach for Multimodal Summarization of TV Shows (2024)

Abstract: In this paper we address the task of summarizing television shows, which touches key areas in AI research: complex reasoning, multiple modalities, and long narratives. We present a modular approach where separate components perform specialized sub-tasks which we argue affords greater flexibility compared to end-to-end methods. Our modules involve detecting scene boundaries, reordering scenes so as to minimize the number of cuts between different events, converting visual information to text, summarizing the dialogue in each scene, and fusing the scene summaries into a final summary for the entire episode. We also present a new metric, PRISMA (**P**recision and **R**ecall Evaluat**i**on of **s**ummary F**a**cts), to measure both precision and recall of generated summaries, which we decompose into atomic facts. Tested on the recently released SummScreen3D dataset (Papalampidi & Lapata, 2023), our method produces higher quality summaries than comparison models, as measured with ROUGE and our new fact-based metric.

## introduction in this paper, we address the challenging task of summarizing television shows which has practical utility in allowing viewers to quickly recall plot points, characters, and events without the need to re-watch entire episodes or seasons. from a computational standpoint, the task serves as a testbed for complex reasoning over long narratives, involving multiple modalities, non-trivial temporal dependencies, inferences over events, and multi-party dialogue with different styles. an added difficulty concerns assessing the quality of generated summaries for long narratives, whether evaluations are conducted by humans or via automatic metrics. most prior work on creative summarization does not consider the above challenges all at once, focusing either on the text modality and full-length narratives with complex semantics @xcite @xcite or on short video clips which last only a couple of minutes @xcite @xcite . a notable exception are @xcite , who incorporate multimodal information into a pre-trained textual summarizer by adding (and tuning) adapter layers @xcite . on the evaluation front, there is no single agreed-upon metric for measuring summary quality automatically, although there is mounting evidence that rouge @xcite does not discriminate between different types of errors, in particular those relating to factuality @xcite . while end-to-end models are a popular choice for summarization tasks @xcite , more modular approaches have been gaining ground recently @xcite @xcite for several reasons. modules can be developed independently, and exchanged for better versions if available, new modules can be added to create new solutions or repurposed for different tasks, and dependencies between modules can be rearranged. aside from greater controllability, modular approaches are by design more interpretable, since errors can be inspected and attributed to specific components. in this paper we delegate the end-toend task of summarizing from multiple modalities (i.e., tv show video and its transcript) to more specialized modules, each responsible for handling different subtasks and their interactions. our approach is depicted graphically in figure 1 . as scene breaks are not always given explicitly, we devise an algorithm to identify them from the order of the speaker names (row 1, figure 1 ). additionally, we select the optimal order in which to re-arrange scenes (row 2, figure 1 ), as these often appear non-linearly (e.g., cuts between different subplots or flashbacks). next, we produce summaries in a two-layer process. a vision-processing emily is accused of hurting rosanna, and she doesn't pass the lie detector test at the police station. at the hospital, ben says he saw aaron the night of the accident and bob speaks to dr. ramirez about it. emily is upset she hurt rosanna. she's afraid of being arrested. emily and hal talk at a police station. emily failed a lie detector test. margo is angry with her. she wants to go home. ## related work in the area of long-form summarization, various methods have been proposed to deal with inputs that exceed the context size of a large language model (llm). memwalker @xcite forms a tree of hierarchical summary nodes and traverses it during inference to find the most relevant parts of the input text. @xcite propose a two-layer method where the top layer captures coarse long-range information and produces topdown corrections to a local attention mechanism in the lower layer. @xcite describe two settings for long-form summarization with llms: hierarchical merging summarizes chunks of the input sequentially, whereas iterative updating con-tinually updates a single summary for each of the chunks. we also adopt a two-layer approach but differ in that we divide the input into semantically meaningful chunks, i.e., scenes, rather than uniform chunks, and we summarize each independently before fusing them together. the problem of generating descriptions for videos has also received much attention, largely independently from long-document summarization. it is common practice @xcite @xcite to extract features from each frame individually, then fuse them into a single feature vector and decode with a language model. swinbert @xcite instead uses an end-toend video network, dispensing with image-based encoders, and samples frames densely. @xcite generate descriptions for short videos with a memory-augmented vision transformer. popular video captioning datasets @xcite are only ∼10s in length. youcook @xcite is a recent dataset with longer videos (5min on average), but still far shorter than the tv shows we focus on here. multimodal summarization is an extension of video captioning in which the input contains text as well as video. @xcite tackle the analogous problem for still-images with a single model whose architecture allows image and text input and can produce a text description as output. @xcite convert the video to text and feed it to a visual storytelling model. @xcite train a vision encoder to produce a sequence of vectors which, when fed to an llm, produce a textual description of the image contents. papalampidi and lapata (2023) apply a similar idea to multimodal summarization, extracting a feature vector from the visual input which is fed, along with the token embeddings from the transcript, to a summarization network. our method does not try to extract a visual feature vector that functions like a token embedding, but rather extracts actual tokens, i.e., a textual description, from the video. ## decomposition into modules our decomposition of the summarization task into modules is motivated by three assumptions specific to tv shows: (1) each scene is somewhat self-contained and, on a coarse level, the events it depicts can be understood independently of other scenes (2) the order in which scenes appear is not necessarily the optimal order to facilitate under-standing, often shows cut back and forth between different plotlines, and sometimes they are presented non-linearly (3) an effective way to capture visual information in a multimodal text summary is to translate it into natural language. these assumptions motivate our modular approach, which is depicted graphically in figure 1 . five separate subtasks are performed by separate components: scene-break detection (top row), scene reordering (second row), converting visual information to text (third row), dialogue summarization (fourth row), and high-level summarization (fifth row). assumption (1) motivates our choice to detect scene breaks and summarize each independently with a module specialized for dialogue summarization and then later fuse these with a high-level summarization module to produce a final output summary. assumption (2) motivates our scenereordering algorithm: we do not simply concatenate scene-level summaries in the order in which they appear, but rather we compute an optimal order designed to minimize the number of transitions between different plotlines. assumption (3) motivates our choice of how to capture the visual information in our summaries. a visual processing module produces a textual description of the video for each scene, which is fed, alongside the dialogue summaries to the high-level summarization module. as a result, the high-level and dialogue summarization modules only need to focus on the single modality of text. ## fact precision, fact recall and prisma hallucinations are a widely known issue with abstractive summarization @xcite @xcite , especially when the output is long @xcite and information is being consolidated from multiple sources @xcite . automated metrics are crucial for our task and related creative summarization applications where human evaluation is extremely labor-intensive (e.g., watching long videos or reading book-length transcripts), costly, and difficult to design @xcite . our proposal is based on a recent metric, factscore @xcite , which aims to detect hallucination in text generation. factscore first parses the generated text into atomic facts (i.e., short sentences conveying one piece of information), and then determines whether these are supported by an external knowledge source, such as wikipedia. the factscore for some model output is the fraction of the extracted facts that are judged to be supported. @xcite recommend using instructgpt for the first stage of converting the text into facts, and llama-7b @xcite for checking whether these facts are supported (i.e., by zero-shot prompting llama to estimate whether a generated fact is true or false). 3 we adapt this for summary evaluation by replacing the external knowledge source with the gold summaries. note that this incorporates relevance as well as accuracy. there are many true facts about the tv show being summarized, and only some of them appear in the gold summaries. we assume that facts that do not appear in the gold summary are irrelevant and so should not be marked as correct if they appear in our model summaries. this is our fact-precision metric. for fact-recall, we do the same in reverse: we convert the gold summary into atomic facts, and then check whether each of these is supported by the generated summary. again, the score is the fraction of such facts that are supported. a summary will get a high fact-precision score if every atomic piece of information is both true and relevant enough to appear in the gold summary. it will get a high fact-recall score if it contains every atomic piece of information that was contained in the gold summary. the final score for our metric, which we term prisma (precision and recall evaluation of summary facts), is the harmonic mean of fact-precision f p and fact-recall f r: in our implementation, we used gpt4-turbo for the extraction of atomic facts and the estimation of whether they are supported, as we found that llama-7b overestimates the number of supported facts. we use the same prompts as @xcite and make a separate query, with in-context learning examples, for each fact. in order to penalize repetitive/redundant information, repeated facts are regarded as unsupported. in the case that gpt indicates that the sentence is not properly formed, we convert it to a single fact which is scored as unsupported (see appendix c for an example). we examined whether prisma correlates with human judgments of factuality. two annotators, both english native speakers, were asked to watch four randomly selected shows from the summ-screen3d development set. they were then shown facts generated by gpt4 corresponding to summaries produced by several automatic systems (those described in section 5.1). for each fact they were asked to decide whether it was supported by the episode (1/true or 0/false). human ratings significantly correlate with gpt4's estimate on whether a fact is supported (pearson's @xmath0.5, @xmath1, @xmath2.01), with an inter-annotator agreement of @xmath3.57 (@xmath4.01) as upper bound. ## experimental evaluation our full model was composed of the modules described in section 3. for those summscreen3d transcripts without explicitly marked breaks (approximately 20%), we split into scenes using algorithm 1. the high-level summarization module was trained for a max of 10 epochs, with early stopping using rouge on the validation set, with a patience of 2. the optimizer was adamw with learning rate 1e-6, and a linear scheduler with 0 warmup steps. training and inference took place on a single nvidia a100-sxm-80gb gpu, taking seven and one hour(s), respectively. the other models in our framework are not fine-tuned. ## conclusion we addressed the task of summarizing tv shows from videos and dialogue transcripts. we proposed a modular approach where different specialized components perform separate sub-tasks. a scenedetection module splits the tv show into scenes, and a scene-reordering module places these scenes in an optimal order for summarization. a dialogue summarization module condenses the dialogue for each scene and a visual processing module produces a textual description of the video contents. finally, a high-level summarization module fuses these into an output summary for the entire episode. we also introduced prisma, a new metric for longform summarization, based on splitting predicted and reference summaries into atomic facts. it captures both precision and recall, and correlates significantly with human judgments. in the future, we plan to test our method on even longer inputs, and explore settings where transcripts are not available.