H ierarchical3 D Adapters for Long Video-to-text Summarization (2023)

Abstract: In this paper, we focus on video-to-text summarization and investigate how to best utilize multimodal information for summarizing long inputs (e.g., an hour-long TV show) into long outputs (e.g., a multi-sentence summary). We extend SummScreen (Chen et al., 2022), a dialogue summarization dataset consisting of transcripts of TV episodes with reference summaries, and create a multimodal variant by collecting corresponding full-length videos. We incorporate multimodal information into a pre-trained textual summarizer efficiently using adapter modules augmented with a hierarchical structure while tuning only 3.8% of model parameters. Our experiments demonstrate that multimodal information offers superior performance over more memory-heavy and fully fine-tuned textual summarization methods.

## introduction what happens in the very last episode of "friends"? anyone who has seen this episode can summarize its key moments: ross confesses his love for rachel, they decide to resume their relationship, while monica and chandler adopt twins and move to the suburbs. tv viewers can naturally perform this dialogue summarization task having access to multiple modalities: they not only hear the actors speak but also see their expressions, actions, and whereabouts on screen. despite recent advances in summarization @xcite @xcite and increasing interest in different types of dialogue summarization, e.g., meeting transcripts @xcite or screenplays @xcite , the contribution of modalities other than text remains relatively understudied. this is not entirely surprising given the challenges associated with the multimodal summarization task * now at deepmind. illustrated above (e.g., produce a written summary of a tv episode). firstly, the input is long, it cannot fit into standard sequence-to-sequence architectures, and the different modalities have to be somehow combined; secondly, the output is also long, summaries consist of multiple sentences and rich vocabulary; and thirdly, it involves complex inference over long-range dependencies between events and characters and common sense reasoning. at the same time, creating large-scale multimodal datasets with long videos and aligned textual data is challenging and time consuming, limiting the research conducted in this domain. previous work on video-to-video summarization identifies highlights from youtube videos, tv shows, or movies @xcite @xcite . however, in most cases, either the videos are short or the datasets are small with a few hundred examples. there is also limited work on videoto-text summarization. we are only aware of one large-scale multimodal dataset for this task, namely how2 @xcite , which again contains short videos (i.e., 2-3 minutes long) with simple semantics, and short, single-sentence summaries. in this paper, we focus on video-to-text summarization and investigate how to best utilize multimodal information for condensing long inputs (e.g., an hour-long tv show) into long outputs (e.g., a multi-sentence summary). we create a multimodal variant of summscreen @xcite , a recently released dataset comprising of transcripts of tv episodes and their summaries. we collect full-length videos for 4,575 episodes and multiple reference summaries. we build our model on top of a pre-trained sequence-to-sequence architecture (i.e., bart; @xcite ) fine-tuned on summarization and capable of generating fluent long text. we convert its textual encoder to a multimodal one by adding and tuning adapter layers @xcite , modality input output datasets text-to-text text short short xsum @xcite , cnn-dailymail @xcite , nyt @xcite , gigaword @xcite text long long samsum @xcite , qmsum @xcite , summscreen @xcite video-to-video vision short short ovp @xcite , youtube @xcite , summe @xcite vision/text short short tvsum @xcite vision/text(/audio) long long lol @xcite tripod+ @xcite ) video-to-text vision long short tacos @xcite ) vision/text/audio short short how2 @xcite ) vision/text/audio long long summscreen 3d our contributions can be summarized as follows: (1) we augment summscreen @xcite with multimodal information, providing videos aligned with transcripts and summaries; to the best of our knowledge, this constitutes the largest available resource for long video multimodal summarization; (2) we propose a parameter efficient approach to augment a pre-trained textual summarizer with multimodal information; and (3) explore different methods for identifying salient moments in a long video and show that multimodal information also improves content selection. ## related work video summarization much previous work has focused on text-to-text or video-to-video summarization. we provide a comprehensive categorization of existing datasets according to input/output length and modality in recent years have also witnessed increasing interest in multimodal video captioning, a task related to multimodal summarization, which aims to generate one-sentence descriptions for localized events in short videos @xcite @xcite . existing methods employ strong language-and-vision encoders with massive pre-training @xcite @xcite @xcite , while the decoder is typically shallow and under-trained. realizing the importance of large lms for generation, recent work has focused on how to efficiently render pre-trained lms multimodal. notably, @xcite convert a pretrained lm into an image captioning model, by giving images as prompts and training only a vision encoder. @xcite summarize how2 videos by augmenting bart with visual information via a new cross-attention block added to every encoder layer. however, their approach adds a very large number of new parameters and requires full fine-tuning, which leads to overfitting in our case when the dataset size is small. ## video-to-text summarization our approach leverages the generation capabilities of large pre-trained sequence-to-sequence models @xcite . as our backbone model, we employ bartlarge @xcite which has been fine-tuned on cnn-dailymail @xcite and has thus acquired a summarization inductive bias. as tv show transcripts are very long and cannot fit into bart, we select a subset of utterances (i.e., speaker turns) as input via content figure 1 : multimodal augmentation of pre-trained bart. we augment the encoder and decoder layers with adapters which we fine-tune on the target dataset, while the remaining network is frozen. as input, we consider textual tokens and coarse-grained multimodal information which we prepend before each utterance. we also corrupt part of the textual input during training and add an auxiliary mlm loss to the encoder for predicting the corrupted tokens. on the right, we show the hierarchical adapter added to each encoder layer: after down-projecting all representations, we only consider the multimodal ones and further contextualize them via attention. then, we combine the representations and up-project again to the original model dimension. ## content selection as explained earlier, episodes in summscreen 3d are very long (∼5,720 tokens). bart, which has a maximum token length of 1,024, can approximately encode one fifth of the transcript. 5 we therefore perform content selection, i.e., identify salient utterances and give these as input to bart. we describe below three approaches inspired by information retrieval, summarization @xcite , and computational narrative analysis @xcite . ## experimental setup implementation details we provide details of the multimodal feature extraction (i.e., utterancelevel visual, audio, and textual features) in appendix b.1. we corrupt the textual input and use the auxiliary emlm loss (section 4.3) only for the first @xmath0,500 training steps; we train our model for a total of 12,000 steps. during corruption, we mask out all content words (i.e., named entities, verbs, and nouns) and a random 10% of the input utterances. for generating summaries during inference, we use beam search with @xmath1. we provide further implementation details in appendix b.2. training vs. inference although we experiment with different content selection methods during inference, we randomly sample input utterances during training. random sampling acts as data augmentation, since the model sees slightly different input-output pairs during training at different iterations. we experimentally verify in section 7 this is preferable to a fixed selection of utterances, especially considering the small size of our dataset. we select @xmath2, and order them according to their original position in the transcript. ## conclusions in this work, we addressed the task of multimodal abstractive summarization and created summscreen 3d , a new dataset which we hope will facilitate future research in this direction. we incorporated multimodal information into a pre-trained textual summarizer in a parameter-efficient manner and have experimentally shown performance gains over text-only models. our experimental results further underscore the importance of (multimodal) content selection compared to approaches focusing on self-attention variants for long dialogue summarization. in the future, we plan to explore more structure-aware representations for all input modalities in order to improve factuality (e.g., entityevent associations) in the generated summaries.