Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization (2021)

Abstract: Multimodal abstractive summarization (MAS) models that summarize videos (vision modality) and their corresponding transcripts (text modality) are able to extract the essential information from massive multimodal data on the Internet. Recently, large-scale generative pre-trained language models (GPLMs) have been shown to be effective in text generation tasks. However, existing MAS models cannot leverage GPLMs’ powerful generation ability. To fill this research gap, we aim to study two research questions: 1) how to inject visual information into GPLMs without hurting their generation ability; and 2) where is the optimal place in GPLMs to inject the visual information? In this paper, we present a simple yet effective method to construct vision guided (VG) GPLMs for the MAS task using attention-based add-on layers to incorporate visual information while maintaining their original text generation ability. Results show that our best model significantly surpasses the prior state-of-the-art model by 5.7 ROUGE-1, 5.3 ROUGE-2, and 5.1 ROUGE-L scores on the How2 dataset, and our vision guidance method contributes 83.6% of the overall improvement. Furthermore, we conduct thorough ablation studies to analyze the effectiveness of various modality fusion methods and fusion locations.

## introduction multimodal abstractive summarization (mas) aims to take advantage of data from multiple modalities and provides a short, concise and readable textual summary to let users quickly acquire their essential information @xcite @xcite . mas has become an increasingly popular research area thanks to the proliferation of online multimedia content and the increasing availability of multimodal data. ## vision guided gplms to take advantage of the superior text generation ability of the text-only seq2seq gplms and adapt them to the mas task, we present vision guided (vg) gplms. specifically, we leverage bart @xcite and t5 @xcite to construct vg-bart and vg-t5. in this section, we start by revisiting the text-only seq2seq gplms in section 3.1. these serve as the backbone of our proposed model and also one of the baselines. then, we discuss the approach for extracting visual features from video clips in section 3.2, as well as how to further process them. finally, in section 3.3, we introduce two types of text-vision fusion mechanism to guide the gplms to generate vision-aware summaries. ## conclusion and future work in this paper, we introduce a simple yet effective method to construct vision guided large-scale generative pre-trained language models (vg-bart and vg-t5) for the multimodal abstractive summarization task by inserting attention-based add-on layers. we propose two types of attention mechanisms for the text-vision fusion and interaction: 1) cross-modal dot-product attention; and 2) crossmodal multi-head attention. moreover, we also transcript: transcript not available summary from transcript + video: learn tips on how to write "cane" in chinese radicals with mandarin characters in the free video clip. get free foreign language lessons from an expert. reference summary: learn what ticks are in chinese calligraphy in this free video clip on languages and writing.