Multimodal Relation Extraction with Cross-Modal Retrieval and Synthesis (2023)

Abstract: Multimodal relation extraction (MRE) is the task of identifying the semantic relationships between two entities based on the context of the sentence image pair. Existing retrieval-augmented approaches mainly focused on modeling the retrieved textual knowledge, but this may not be able to accurately identify complex relations. To improve the prediction, this research proposes to retrieve textual and visual evidence based on the object, sentence, and whole image. We further develop a novel approach to synthesize the object-level, image-level, and sentence-level information for better reasoning between the same and different modalities. Extensive experiments and analyses show that the proposed method is able to effectively select and compare evidence across modalities and significantly outperforms state-of-the-art models.

## introduction relation extraction aims to detect relations among entities in the text and plays an important role in various applications @xcite . early efforts mainly focus on predicting the relations based on the information from one single modality i.e., text. recently, multimodal relation extraction (mre) has been proposed to enhance textual representations with the aid of visual clues from images @xcite @xcite . it extends the textbased approaches by providing visual contexts to address the common ambiguity issues in identifying relations. figure 1 shows an example from the mnre dataset @xcite . to infer the relation between entities ang lee and oscar, the model needs to capture the interactions from visual relations between objects in an image to textual relations in a sentence. the visual relation "holding" forget the dresses, ang lee [per] is my favorite oscar [misc] actor. between two objects helps to detect the relation awarded between two textual entities. most existing efforts focus on modeling the visual and textual content of the input. zheng et al. (2021a) constructed the textual and visual graphs, then identify the relations based on graph alignments. chen et al. (2022) presents a hierarchical visual prefix fusion network to incorporate hierarchical multi-scaled visual and textual features. li et al. (2023a) proposes a fine-grained multimodal alignment approach with transformer, which aligns visual and textual objects in representation space. wang et al. (2022) first proposes retrieval-augmented multimodal relation extraction. the given image and sentence are used to retrieve textual evidence from the knowledge base constructed based on wikipedia. unlike previous retrieval-based models, we not only retrieve texts but also retrieve visual and textual evidence related to the object, sentence, and entire image. a novel strategy is used to combine evidence from the ob- ## related work relation extraction has garnered considerable interest in the research community due to its essential role in various natural language processing applications @xcite @xcite . the initial efforts in this field focused on detecting relations between entities in the text, with different neural architectures @xcite @xcite and pretrained language models @xcite used to encode the textual informa-tion. multimodal relation extraction has recently been proposed, where visual clues from images are used to enhance entity representations @xcite @xcite . most existing efforts focus on fusing the visual and textual modalities effciently. zheng et al. (2021b) constructed the dual modality graph to align multimodal features among entities and objects. chen et al. (2022) concatenated object-level visual representation as the prefix of each self-attention layer in bert. li et al. (2023a) introduced a fine-grained multimodal fusion approach to align visual and textual objects in representation space. closest to our work, wang et al. (2022) proposed to retrieve textual information related to the entities based on the given image and sentence. unlike prior efforts, we not only retrieve texts related to entities but also retrieve visual and textual evidence related to the object, sentence, and entire image. we further synthesize the retrieved object-level, image-level, and sentence-level information for better reasoning between the same and different modalities. 5 conclusion and future work we propose to retrieve multimodal evidence and model the interactions among the object, sentence, and whole image for better relation extraction. experiments show that the proposed method achieves competitive results on mnre. for future research directions, we can utilize open-source image search and caption generation tools to retrieve textual and image evidence. for example, to retrieve visual evidence, one can (1) use a web crawler to search google images, or (2) utilize a searchable image database: pigallery foot_2 , where images can be sourced from open image dataset foot_3 , which contains ∼9 million images. for retrieving textual evidence, one can use clip to generate image captions. moreover, we can also apply the method of multimodal retrieval to low-resource relation extraction @xcite @xcite , natural language inference @xcite , semantic parsing @xcite , and other nlp tasks, thus realizing information enhancement based on images and retrieval. in this paper, we suggest incorporating textual and visual data from search engines for multimodal relation extraction. despite the fact that the proposed model yields competitive results on the benchmark, it still has several limitations. firstly, using a search engine is a feasible way to obtain related knowledge, but it also brings the issue of noisy evidence. unrelated visual and textual evidence returned by the search engine may lead to incorrect predictions from the model. additionally, not all the retrieved evidence is equally reliable, and sometimes sources may contradict each other. on the other hand, retrieval-augmented methods are slower than content-based counterparts, since retrieving evidence from the internet requires extra time. therefore, it may not satisfy some of the time-sensitive scenarios. lastly, evidence may be presented in different forms other than texts and images. for instance, structural information such as tables, info lists, and knowledge graphs also provide important contexts for identifying semantic relations. humans are able to extract relevant information from these heterogeneous sources for inference, while our relation extraction system can only model and reason over textual and visual evidence.