Leveraging Entity Information for Cross-Modality Correlation Learning: The Entity-Guided Multimodal Summarization (2024)

Abstract: The rapid increase in multimedia data has spurred advancements in Multimodal Summarization with Multimodal Output (MSMO), which aims to produce a multimodal summary that integrates both text and relevant images. The inherent heterogeneity of content within multimodal inputs and outputs presents a significant challenge to the execution of MSMO. Traditional approaches typically adopt a holistic perspective on coarse image-text data or individual visual objects, overlooking the essential connections between objects and the entities they represent. To integrate the fine-grained entity knowledge, we propose an Entity-Guided Multimodal Summarization model (EGMS). Our model, building on BART, utilizes dual multimodal encoders with shared weights to process text-image and entity-image information concurrently. A gating mechanism then combines visual data for enhanced textual summary generation, while image selection is refined through knowledge distillation from a pre-trained vision-language model. Extensive experiments on public MSMO dataset validate the superiority of the EGMS method, which also prove the necessity to incorporate entity information into MSMO problem.

## introduction with the rapid development of multimedia content across the internet, the task of multimodal summarization with multimodal output (msmo) has emerged as a research direction of considerable significance @xcite @xcite , especially for news content summary @xcite . specifically, as shown in figure 1 , given the source text and corresponding images, msmo aims to produce a multimodal summary with a textual abstract alongside a pertinent image. instead of providing exclusively text-based summaries, msmo considers and generates more diverse multimodal information, which * corresponding author. ## conclusions in this paper, we propose an entity-guided multimodal summarization model (egms), that incorporates entity-specific information into solving msmo problem. based on bart, our model introduces a pair of multimodal encoders with shared weights to concurrently process text-image and entity-image information. subsequently, a gating mechanism is used to fuse the visual representations, which will further be utilized in the generation of textual summaries. as for image selection, we also use a gating mechanism and distill knowledge from a pre-trained vision-language model. extensive experiments on public msmo dataset demonstrat the effectiveness of our proposed method. we hope our work could lead to more future studies in this field.