Multimodal Sentence Summarization via Multimodal Selective Encoding (2020)

Abstract: This paper studies the problem of generating a summary for a given sentence-image pair. Existing multimodal sequence-to-sequence approaches mainly focus on enhancing the decoder by visual signals, while ignoring that the image can improve the ability of the encoder to identify highlights of a news event or a document. Thus, we propose a multimodal selective gate network that considers reciprocal relationships between textual and multi-level visual features, including global image descriptor, activation grids, and object proposals, to select highlights of the event when encoding the source sentence. In addition, we introduce a modality regularization to encourage the summary to capture the highlights embedded in the image more accurately. To verify the generalization of our model, we adopt the multimodal selective gate to the text-based decoder and multimodal-based decoder. Experimental results on a public multimodal sentence summarization dataset demonstrate the advantage of our models over baselines. Further analysis suggests that our proposed multimodal selective gate network can effectively select important information in the input sentence.

## introduction text summarization is a task that condenses a long sentence to a short version. existing researches @xcite @xcite @xcite @xcite ) produce summary only from the text. however, it has been proved that human understands the text relying on multimodal information @xcite @xcite , such as linguistic and visual signals. in this paper, we focus on the multimodal summarization task @xcite ) that generates summary simultaneously drawing knowledge from coupled text and image, which can facilitate other applications such as image captioning @xcite , multimodal news summarization @xcite @xcite , and electronic commerce (e-commerce) product description generation @xcite @xcite , etc. intuitively, it is easier for a reader to grasp the highlight of a news event by viewing an image than by reading a long text. hence we believe that the image will benefit text summarization system. figure 1 illustrates this phenomenon. for a given source sentence, a paired image visualizes a set of event highlight words, which highly correlates with the reference summary. multimodal sequence-to-sequence (seq2seq) learning has been widely explored in machine translation (mt) @xcite @xcite in recent years, and the performances of their models surpass text-only models @xcite . the main difference between multimodal mt and multimodal text summarization is: the model for mt is required to convert the same semantics from the input sentence and the paired image to the output, while for summarization, the model is expected to select the important information from the input. @xcite propose a hierarchical attention model for the multimodal sentence summarization task, while the image is not involved in the process of text encoding. obviously, it will be easier for the decoder to generate this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . source sentence: british prime minister tony blair has rescued a danish swimmer from the shark infested sea during his holiday in seychelles in africa , a government spokesman said friday . reference summary: uk pm rescues swimmer from sea . an accurate summary if the encoder can filter out trivial information when encoding the input sentence. based on this idea, we propose a multimodal selective mechanism which aims to select the highlights from the input text using visual signals, and then the decoder generates the summary using the filtered encoding information. concretely, an encoder reads the input text and generates the hidden representations. then, multimodal selective gates measure the relevance between the input words and the image to construct the selected hidden representation. finally, a decoder generates the summary using the selected hidden representation. we explore the capacity of the image to select salient content from the input text at different levels including global image descriptor, activation grids, and object proposals in the image. accordingly, as shown in figure 2 , we design three visual selective gates: global-level, grid-level, and object-level gates. furthermore, some abstract concepts, such as "guilty" and "freedom", can hardly be well represented by the image, and thus we combine textual and visual selective gates to construct multimodal selective gates. in addition, we argue that a good summarizer should adequately capture the highlight words. thus, we impose a modality regularization on our model that encourages the semantic similarity between (image, generated summary) higher than that between (image, source text). our main contributions are as follows: ## conclusion this paper addresses the multimodal text summarization task, namely, transforming a pair of text and image into a summary. we propose multimodal selective encoding gates that can effectively control the information flow from the encoder to the decoder. we adopt our multimodal selective encoding strategies to the text-based and multimodal-based decoders. the experimental results on the public multimodal sentence summarization corpus prove that our proposed model significantly outperforms other comparative methods. note that our method is not specifically designed for rnn-based models, and it can be easily applied to transformer-based models, which is left for our future work.