Exploiting Pseudo Image Captions for Multimodal Summarization (2023)

Abstract: Multimodal summarization with multimodal output (MSMO) faces a challenging semantic gap between visual and textual modalities due to the lack of reference images for training. Our pilot investigation indicates that image captions, which naturally connect texts and images, can significantly benefit MSMO. However, exposure of image captions during training is inconsistent with MSMO’s task settings, where prior cross-modal alignment information is excluded to guarantee the generalization of cross-modal semantic modeling. To this end, we propose a novel coarse-to-fine image-text alignment mechanism to identify the most relevant sentence of each image in a document, resembling the role of image captions in capturing visual knowledge and bridging the cross-modal semantic gap. Equipped with this alignment mechanism, our method easily yet impressively sets up state-of-the-art performances on all intermodality and intramodality metrics (e.g., more than 10% relative improvement on image recommendation precision). Further experiments reveal the correlation between image captions and text summaries, and prove that the pseudo image captions we generated are even better than the original ones in terms of promoting multimodal summarization.

## introduction with the increase of multimedia data on the web, multimodal summarization has drawn widespread attention from researchers in the communities of web technologies @xcite , natural language processing (nlp) @xcite @xcite and computer vision (cv) @xcite @xcite . more recently, many efforts @xcite @xcite have been dedcompared with text summarization models, existing msmo methods usually use an extra image encoder to project images into intermediate representations. they identify the salient image by cross-modal attention, which could be inaccurate due to the lack of golden images for training. we explicitly transform an image into a concrete caption by image-text alignment, capturing visual knowledge better and making text summarization and image selection more effective yet simpler. msmo faces two main challenges. (1) there are no recommended image references available for training msmo models. due to the lack of supervision signals from visual modality, it is nontrivial to optimize the cross-modal attention between texts and images, which is highly relied on by existing msmo methods to pick salient images. according to previous best results @xcite , only about 60% of the predicted images are correct, indicating that image selection remains a bottleneck. (2) visual knowledge is commonly underutilized to improve text summaries. existing msmo efforts show no evident improvement or even negative impact on text summaries (e.g., decreased rouge scores) over typical singlemodal text summarization methods. previous literature @xcite explained that some images were noises and long text had contained enough information for text generation, while we conjecture that these methods may not sufficiently exploit visual knowledge to characterize salient text. to summarize, previous efforts typically encode images and texts into the same semantic space, struggling with optimizing cross-modal interaction without training signals for image selection, as the red box in figure 1 shows. in this dilemma, image captions, which naturally connect images and texts, can provide a cross-modal semantic bridge. indeed, our preliminary experiments show the efficacy of introducing imagein captions (see section 4.4). yet, exposure of image captions during training is inconsistent with msmo's task settings, since msmo excludes them to pursue better generalization of cross-modal semantic modeling @xcite .on the other hand, however, it inspires us to identify a highly-relevant sentence for an image as its pseudo yet meaningful caption, providing us with a new perspective to improve msmo. as shown in the blue box in figure 1 , unlike current works that represent an image as an intermediate state, we transform it into a concrete sentence to better capture visual knowledge under msmo settings. this transformation presents an opportunity to incorporate pre-trained visual-language models more smoothly, while making further text summarization and image selection extremely simple. aligning a sentence with an image could be straightforward, but identifying sentences benefiting msmo the most is non-trivial. the reasons are two-fold. (1) a sentence well aligned with an individual image can not guarantee a suitable one for msmo. an intuitive way to select a sentence is to simply retrieve it from the document, with the image as the query of a pre-trained cross-modal retrieval model. unfortunately, we find this manner yields unsatisfactory msmo performance (see section 4.3). (2) a classical single-pass one-to-one alignment strategy may miss salient sentences for summarization (see section 4.2). there can be one-to-many and many-to-one relationships between images and sentences, and images can be similar in a document, so we need to synthesize yet distinguish image semantics from a global perspective to make better msmo-oriented alignment. to this end, we design a coarse-to-fine imagetext alignment mechanism to produce pseudo image captions for msmo. firstly, a reference caption for an image is retrieved with a cross-modal retrieval model from the golden summary, rather than the whole document (section 2.3), to capture more summary-friendly information. since no golden summary exists at inference time, these reference captions are used to train a two-pass image-text alignment model (section 2.4) that yields pseudo captions when making inferences (that's why "reference captions" are so named). given a document with ten images, for example, we will first synthesize them as a whole to select ten sentences with many-to-many coarse-grained alignment, and then identify ten individual one-to-one fine-grained matchings by bipartite graph matching over the cross-modal attention matrix. the pseudo image captions that imply visual knowledge are used as extra highlighted features for text summarization (section 2.5), and the salient image is picked based on the rouge score between its pseudo captions and the generated summary (section 2.6). extensive experiments on an existing msmo dataset not only verify the superiority of our method but also reveal the inner connection between image captions and summaries, demonstrating promising research opportunities for our novel perspective of bridging the cross-modal semantic gap by generating pseudo image captions. ## related work existing text summarization approaches can be roughly categorized into extractive summarization @xcite @xcite and abstractive summarization @xcite @xcite @xcite . classical abstractive summarizaition model such as pointer generator network @xcite and bertsum @xcite serve as fundamental components for previous msmo works. multimodal summarization takes data of more than one modalities as input and synthesizes information across different modalities to generate the output @xcite @xcite @xcite @xcite @xcite @xcite . @xcite first propose generating pictorial summaries given a document and an image collection. @xcite further introduced a extra cross-entropy loss for image selection. recently, @xcite proposed to utilize knowledge distillation with a vision-language pre-trained model to help image selection, but the image precision was still far from ideal. ## conclusion we have presented sita, a multimodal summarization method based on coarse-tofine image-text alignment. sita introduces a novel perspective of bridging the semantic gap between visual and textual modality by exploiting pseudo image captions. our cross-modal alignment mechanism effectively generates high-quality pseudo image captions, enabling sita to set up state-of-the-art performance easily. we discuss the feasibility and potential of leveraging pseudo image captions , and release code 2 , to inspire more future studies from our proposed perspective.