CFS um Coarse-to-Fine Contribution Network for Multimodal Summarization (2023)

Abstract: Multimodal summarization usually suffers from the problem that the contribution of the visual modality is unclear. Existing multimodal summarization approaches focus on designing the fusion methods of different modalities, while ignoring the adaptive conditions under which visual modalities are useful. Therefore, we propose a novel Coarse-to-Fine contribution network for multimodal Summarization (CFSum) to consider different contributions of images for summarization. First, to eliminate the interference of useless images, we propose a pre-filter module to abandon useless images. Second, to make accurate use of useful images, we propose two levels of visual complement modules, word level and phrase level. Specifically, image contributions are calculated and are adopted to guide the attention of both textual and visual modalities. Experimental results have shown that CFSum significantly outperforms multiple strong baselines on the standard benchmark. Furthermore, the analysis verifies that useful images can even help generate non-visual words which are implicitly represented in the image.

## introduction with the information explosion, the internet is flooded with various multimodal information. multimodal summarization (mms) can help generate more abundant and comprehensive summary information than unimodal based on extra visual information. existing studies on multimodal summarization include multimodal sentence summarization @xcite , multimodal summarization with figure 1 : experiments on existing mainstream multimodal summarization models. the performance is not affected by masking images. "concat" is the concatenate fusion method, and "attn" is the attention-based fusion method. multimodal output @xcite , multimodal meeting summarization @xcite and so on. in this paper, we focus on the task that generating a text summary based on the input of a text and an image. it has been proved that integrating multimodal data can help improve the quality of the summary @xcite @xcite . however, it is unclear whether the visual modality can indeed benefit the process of summarization. thus, we conduct an experiment to explore the influence of masking images on the summary. as shown in figure 1 , the solid lines mean the performance of summary generated by masking portions of images, and the dashed lines indicate the origin performance. it can be observed that the dashed and the solid lines roughly coincide, which indicates that masking images do not affect the performance of the multimodal model. some masking rates can even raise the rouge-1 value of the summary. it indicates that existing models do not make effective use of image information for the summary. existing approaches have two major problems. first, existing studies focus on multimodal fusion, such as concatenate, attention-based, and gatebased fusion (referring to related work). however, they ignore the adaptive conditions under which visual modalities are helpful. thus they are poor at extracting useful visual information. furthermore, all fusion methods do not explicitly model the image complementarity for the summary. especially for the attention-based method, the inter-attention is not accurate enough, which leads to inefficient use of the image. second, in many samples, the image may introduce noise, while existing fusion methods assume that all images are helpful for the summary without considering the interference of useless images. as analyzed above, we believe that: although we notice the lack of image contributions, it is difficult to detach various roles of images from a single fusion layer. thus, in this work, we propose a novel coarse-to-fine contribution network for multimodal summarization (cfsum) to extract the role of the image at different stages. first, we apply a pre-filter module to abandon useless images. it coarsely obtains helpful images for the summary. specifically, the consistency of content between image and text is calculated. if the consistency is low, the image will be masked in subsequent encoding. second, when the image is coarsely useful, the complement module is employed to finely guide the fusion of text with the image. to consider image contributions for text with different granularities, the complement module consists of two levels, word level and phrase level. for the word level complement module, to obtain the image complementarity over the text, the difference between bi-modal and uni-modal inputs is measured through a classification task. then we add a loss to guide the attention between words and the image. for the phrase level complement module, similar to the word level, the image complementarity on phrases is acquired to guide the attention between phrases and the image. through these modules, the model can acquire more explicit image contributions and provide better multimodal encoding for summary generation. our contributions are as follows: ## related work multimodal summarization tasks. in the field of multimodal summarization, there are usually three steps. first, different feature extractor modules are adopted to extract the features of the text and the image, respectively. second, the different features are fused at the fusion layer. finally, the fused context features are fed into the text decoder to generate a summary. existing studies focus on multimodal fusion. specifically, the fusion methods consist of concatenate, attention-based, and gate-based. the concatenate fusion directly concatenates multimodal features into a fusion context @xcite . it can fully extract highlevel features of different modalities, but there is a large gap between high-dimensional spaces. the attention-based methods fuse all multimodal features with attention mechanism @xcite @xcite , which can get the correlations between each unit of text and image. gate-based methods take text as the central modality @xcite and exploit images to help focus on the core information @xcite . in summary, (1) all fusion methods do not explicitly model the image complementarity for the summary, which leads to inefficient use of the image. (2) concatenate and attention-based cannot eliminate the influence of useless images in the fusion layer. cross-modal tasks. some studies have noted the contributions of modalities and explored the crossmodal influence in other multimodal tasks. @xcite propose loss modulation to explore the contribution of individual modalities and devise a modality filter to reduce modality noise, which considers consistency and complementarity between different modalities. zhu et al. (2018) propose multi-task summarization: the method also selects the image that best matches the summary when generating a text summary. it guarantees the positive effect of images on the summary. li et al. (2022) exploit relu-based cross-attention to align visual features to textual representation, which abandons low-value attention scores for those unaligned visual features. inspired by the above studies, we propose cfsum, which considers various image contributions for better encoding input text and generating the final summary. ## conclusion based on the observation that existing multimodal summary models do not take full advantage of useful image information, this paper focuses on modeling different contributions of images for summarization. therefore, we propose a novel framework cfsum consisting of pre-filter, word-level complement, and phrase-level complement modules. the pre-filter coarsely eliminates the impact of useless images. the two-level visual complement modules measure different aspects of image gains and guide the fusion of different modalities. experimental results have shown that cfsum can significantly improve the summary. more importantly, the complement modules make images contribute to visual words and non-visual words.