Summary-Oriented Vision Modeling for Multimodal Abstractive Summarization (2023)

Abstract: The goal of multimodal abstractive summarization (MAS) is to produce a concise summary given the multimodal data (text and vision). Existing studies on MAS mainly focus on how to effectively use the extracted visual features, having achieved impressive success on the high-resource English dataset. However, less attention has been paid to the quality of the visual features to the summary, which may limit the model performance, especially in the low- and zero-resource scenarios. In this paper, we propose to improve the summary quality through summary-oriented visual features. To this end, we devise two auxiliary tasks including vision to summary task and masked image modeling task. Together with the main summarization task, we optimize the MAS model via the training objectives of all these tasks. By these means, the MAS model can be enhanced by capturing the summary-oriented visual features, thereby yielding more accurate summaries. Experiments on 44 languages, covering mid-high-, low-, and zero-resource scenarios, verify the effectiveness and superiority of the proposed approach, which achieves state-of-the-art performance under all scenarios. Additionally, we will contribute a large-scale multilingual multimodal abstractive summarization (MM-Sum) dataset to the research community.

## introduction given an article and several images as inputs, as shown in fig. 1 , multimodal abstractive summarization (mas) @xcite @xcite @xcite aims to generate a concise textual summary, which can help people quickly grasp the core information. therefore, mas has widespread application and figure 1 : an example of our mm-sum dataset. inputs: an article and image sequence pair; output: summary. as we can see, the image sequence also concisely paraphrases the summary. the red content indicates its associated object is useless to the summary while the green counterparts represent important information. recently, many studies have been carried out to effectively inject the visual features into mas models @xcite @xcite @xcite @xcite . for instance, @xcite and @xcite explore the hierarchy between the textual article and visual features, and integrate them into the mas model. @xcite design a multistage fusion network to model the fine-grained interactions between the two modalities. and @xcite study multiple multimodal fusion methods to infuse the visual features into generative pre-trained language models, e.g., bart @xcite . despite their success on the high-resource english dataset, they only model visual features from the perspective of an article and neglect the relevance of visual features to the summary, which restricts their potential performance especially on the training dataset with limited scale. for example, though the object "black clothes" in the first image of fig. 1 is associated with the article content (red part), the object contributes little to the summary. thus, the mas model should focus on summary-oriented visual features. however, the visual features are generally implicitly learned via the mas objective, which cannot help the model learn to explicitly discard the needless visual information. to address this issue, in this paper, we propose a summary-oriented vision enhanced mas (sov-mas) training framework to generate more accurate summaries through explicitly improving the relevance of visual features to the summary. to this end, we design two summary-oriented vision modeling tasks, namely vision to summary task, and masked image modeling task. specifically, as shown in fig. 2 , (1) the vision to summary task is to produce the concise summary by only taking the image sequence; (2) the masked image modeling task aims to predict the semantic class distribution of the regions in one fully masked image given the summary and the remaining images. together with the main multimodal summarization task, the mas model is optimized through the joint objectives of all these tasks. in this way, the model is enhanced to explicitly exploit the summary-oriented visual features, thus leading to more accurate summaries. to validate the sov-mas framework on various languages and diverse settings, we construct the first large-scale multilingual multimodal summarization dataset (mm-sum) based on xl-sum @xcite , a multilingual summarization dataset. the mm-sum covers 44 languages with mid-high-, low-and zero-resource scenarios. experiments on these settings show that our model significantly outperforms related methods in terms of rouge @xcite ) scores, especially under the low-and zero-resource settings, demonstrating its effectiveness. besides, we extend our approach to two previous best mas models (i.e., vg-bart and vg-t5 @xcite ). human evaluation and the results on how2 @xcite benchmark further suggest the superiority and generalizability of our approach. in summary, our main contributions are: ## sov-mas framework based on the vision-guided pre-trained language model described in § 2.2, we introduce the proposed summary-oriented vision enhanced mas ((sov-mas)) framework. specifically, we firstly describe the process of visual features extraction in § 3.1. then, to make the best use of visual features, we design two summary-oriented vision modeling tasks in § 3.2, namely vision to summary task and masked image modeling task. finally, we describe the training and inference in § 3.3. ## related work abstractive text summarization (ats). given the input textual article, the goal of ats is to generate a concise summary @xcite . thanks to generative pretrained language models @xcite , ats has achieved remarkable performance @xcite @xcite 9.0 points, respectively) ## conclusion in this paper, we propose to enhance the mas model through two summary-oriented vision modeling tasks namely vision to summary task and masked image modeling task. they can explicitly force the mas model to exploit the summaryoriented visual features and thus improve the summary quality. extensive experiments on multiple settings demonstrate that our model significantly outperforms related baselines in terms of rouge scores and human evaluation. furthermore, we contribute a large-scale multilingual mas (mm-sum) dataset to the research community.