VMSMO : Learning to Generate Multimodal Summary for Video-based News Articles (2020)

Abstract: A popular multimedia news format nowadays is providing users with a lively video and a corresponding news article, which is employed by influential news media including CNN, BBC, and social media including Twitter and Weibo. In such a case, automatically choosing a proper cover frame of the video and generating an appropriate textual summary of the article can help editors save time, and readers make the decision more effectively. Hence, in this paper, we propose the task of Video-based Multimodal Summarization with Multimodal Output (VMSMO) to tackle such a problem. The main challenge in this task is to jointly model the temporal dependency of video with semantic meaning of article. To this end, we propose a Dual-Interaction-based Multimodal Summarizer (DIMS), consisting of a dual interaction module and multimodal generator. In the dual interaction module, we propose a conditional self-attention mechanism that captures local semantic information within video and a global-attention mechanism that handles the semantic relationship between news text and video from a high level. Extensive experiments conducted on a large-scale real-world VMSMO dataset show that DIMS achieves the state-of-the-art performance in terms of both automatic metrics and human evaluations.

## introduction existing experiments @xcite have proven that multimodal news can significantly improve users' sense of satisfaction for informativeness. as one of these multimedia data forms, introducing news events with video and textual descriptions is becoming increasingly popular, and has been employed as the main form of news reporting by news media including bbc, weibo, cnn, and daily mail. an illustration is shown in figure 1 , where the news contains a video with a cover picture and a full news article with a short textual summary. in such a case, automatically generating multimodal summaries, i.e., choosing a proper cover frame of the video and generating an appropriate textual summary of the article can help editors save time and readers make decisions more effectively. there are several works focusing on multimodal summarization. the most related work to ours is @xcite , where they propose the task of generating textual summary and picking the most representative picture from 6 input candidates. however, in real-world applications, the input is usually a video consisting of hundreds of frames. consequently, the temporal dependency in a video cannot be simply modeled by static encoding methods. hence, in this work, we propose a novel task, video-based multimodal summarization with multimodal output (vmsmo), which selects cover frame from news video and generates textual summary of the news article in the meantime. the cover image of the video should be the salient point of the whole video, while the textual summary should also extract the important information from source articles. since the video and the article focus on the same event with the same report content, these two information formats complement each other in the summarizing process. however, how to fully explore the relationship between temporal dependency of frames in video and semantic meaning of article still remains a problem, since the video and the article come from two different space. hence, in this paper, we propose a model named dual-interaction-based multimodal summarizer (dims), which learns to summarize article and video simultaneously by conducting a dual interaction strategy in the process. specifically, we first employ recurrent neural networks (rnn) to encode text and video. note that by the encoding rnn, the spatial and temporal dependencies between images in the video are captured. next, we design a dual interaction module to let the video and text fully interact with each other. specifically, we propose a conditional self-attention mechanism which learns local video representation under the guidance of article, and a global-attention mechanism to learn high-level representation of videoaware article and article-aware video. last, the multimodal generator generates the textual summary and extracts the cover image based on the fusion representation from the last step. to evaluate the performance of our model, we collect the first large-scale news article-summary dataset associated with video-cover from social media websites. extensive experiments on this dataset show that dims significantly outperforms the state-of-the-art baseline methods in commonly-used metrics by a large margin. to summarize, our contributions are threefold: ## related work our research builds on previous works in three fields: text summarization, multimodal summarization, and visual question answering. text summarization. our proposed task bases on text summarization, the methods of which can be divided into extractive and abstractive methods @xcite . extractive models @xcite @xcite @xcite directly pick sentences from article and regard the aggregate of them as the summary. in contrast, abstractive models @xcite @xcite @xcite @xcite generate a summary from scratch and the abstractive summaries are typically less redundant. multimodal summarization. a series of works @xcite @xcite @xcite focused on generating better textual summaries with the help of multimodal input. multimodal summarization with multimodal output is relatively less explored. @xcite proposed to jointly generate textual summary and select the most relevant image from 6 candidates. following their work, @xcite added a multimodal objective function to use the loss from the textual summary generation and the image selection. however, in the real-world application, we usually need to choose the cover figure for a continuous video consisting of hundreds of frames. consequently, the temporal dependency between frames in a video cannot be simply modeled by several static encoding methods. visual question answering. visual question answering (vqa) task is similar to our task in taking images and a corresponding text as input. most works consider vqa task as a classification problem and the understanding of image subregions or image recognition becomes particularly important @xcite @xcite . as for the interaction models, one of the state-of-theart vqa models @xcite proposed a positional self-attention with a co-attention mechanism, which is faster than the recurrent neural network (rnn). @xcite devised an image- frame encoder segment encoder conditional self-attention input article globalattention fusion gates editing gate decoder generated summary matching score selected cover feature encoder dual interaction module multi-generator input video frames text encoder figure 2: overview of dims. we divide our model into three parts: (1) feature encoder encodes the input article and video separately; (2) dual interaction module learns fused representation of video and article from different level; (3) multi-generator generates the textual summary and chooses the video cover simultaneously. ## problem formulation before presenting our approach for the vmsmo, we first introduce the notations and key concepts. for an input news article @xmath0, x 2 , . . . , x t d } which has t d words, we assume there is a ground truth textual summary @xmath1, y 2 , . . . , y ty } which has t y words. meanwhile, there is a news video v corresponding to the article, and we assume there is a ground truth cover picture c that extracts the most important frame from the video content. for a given article x and the corresponding video v , our model emphasizes salient parts of both inputs by conducting deep interaction. the goal is to generate a textual summary y that successfully grasp the main points of the article and choose a frame picture c that covers the gist of the video. ## conclusion in this paper, we propose the task of video-based multimodal summarization with multimodal output (vmsmo) which chooses a proper video cover and generates an appropriate textual summary for a video-attached article. we propose a model named dual-interaction-based multimodal summarizer (dims) including a local conditional self-attention mechanism and a global-attention mechanism to jointly model and summarize multimodal input. our model achieves state-of-the-art results in terms of autometrics and outperforms human evaluations by a large margin. in near future, we aim to incorporate the video script information in the multimodal summarization process.