Visual Enhanced Entity-Level Interaction Network for Multimodal Summarization (2024)

Abstract: MultiModal Summarization (MMS) aims to generate a concise summary based on multimodal data like texts and images and has wide application in multimodal fields.Previous works mainly focus on the coarse-level textual and visual features in which the overall features of the image interact with the whole sentence.However, the entities of the input text and the objects of the image may be underutilized, limiting the performance of current MMS models.In this paper, we propose a novel Visual Enhanced Entity-Level Interaction Network (VE-ELIN) to address the problem of underutilization of multimodal inputs at a fine-grained level in two ways.We first design a cross-modal entity interaction module to better fuse the entity information in text and the object information in vision.Then, we design an object-guided visual enhancement module to fully extract the visual features and enhance the focus of the image on the object area.We evaluate VE-ELIN on two MMS datasets and propose new metrics to measure the factual consistency of entities in the output.Finally, experimental results demonstrate that VE-ELIN is effective and outperforms previous methods under both traditional metrics and ours.The source code is available at https://github.com/summoneryhl/VE-ELIN.

## introduction multimodal summarization (mms) takes multimodal data like texts and images as input and aims to generate a concise summarization as output. this task has attracted much attention in the research community @xcite @xcite because it can be widely used in various real-world applications, such as social media @xcite , meeting @xcite , and e-commerce products @xcite . recent studies primarily concentrate on the cross-modal interaction and filtering of visual features, which have achieved promising performances. for instance, @xcite explores various ways of image-text fusion to utilize multimodal information based on the application of generative pre-trained language models (plms) to the task. @xcite adopts knowledge distillation from the vision-language pre-trained model to improve image selection. @xcite designs a target-oriented contrastive objective to discard needless visual information. despite their effectiveness, current methods mainly focus on the coarse-level rather than fine-grained visual and textual features, which conduct interactions between the global image and sentence semantics. this might lead to an insufficient utilization of crucial local information. as shown in figure 1 , there are three fine-grained entities "nicole cooke", "gold", and "beijing olympics" in the input text, and three object regions in the image corresponding to them while previous methods are not able to extract the fine-grained information adequately. thus, we consider utilizing the inherent entity information in the text and object information in the image so that the output summary maintains key entities with high coherence. in this paper, we propose a novel visual enhanced entity-level interaction network (ve-elin) for multimodal summarization. the proposed ve-elin addresses the problem of incomplete generation of entity information in two ways. firstly, we design the crossmodal entity interaction (ei) module which can better fuse the entity information in text and the object information in vision and provide richer multimodal representation. in particular, the ei module includes three levels of features, namely sentence, entity, and object level. we encode the input text using a textual encoder to obtain sentence-level features and use a pre-trained named entity recognition model @xcite to get entity-level features. moreover, we use the image object detection model @xcite to capture the objects in the image and encode them to obtain the objectlevel features. secondly, to further distill features from vision information, we apply clip @xcite and integrate it into our object-guided visual enhancement (ve) module. the ve module can fully extract the visual features and enhance the focus of the image on the object area to better inject visual information into the multimodal decoder. in addition to conventional evaluation methods, we introduce novel metrics to measure the factual consistency of entities in the output summarization. specifically, we count the number of entities in the output and compare it with the entities in the target summary. then, we compute the proportion of entities named entityscore and the similarity between entities named similarscore. we evaluate ve-elin on two mms datasets, which have different text lengths and input image numbers. the experimental results demonstrate that ve-elin is effective and outperforms previous methods under both traditional metrics and ours. in summary, our contributions are as follows: ## methodology in this section, we introduce the overview of our framework. we first present the brief task formulation and describe the method overview. then, we detail our proposed module and introduce the training and generation process. ## conclusion in this paper, we propose a novel framework ve-elin for multimodal summarization to alleviate the incomplete generation of entity information in summary. we design a cross-modal entity interaction module to better utilize the entity features in texts and images, and an object-guided visual enhancement module to enhance the focus on the objects while taking full advantage of useful image information. to further evaluate the factual consistency of entities in the output summary, we also propose two new metrics named entityscore and similarscore. experimental results on two different types of datasets demonstrate that our method is effective and outperforms previous methods under both traditional evaluation metrics and our proposed new metrics.