Dual-Gated Fusion with Prefix-Tuning for Multi-Modal Relation Extraction (2023)

Abstract: Multi-Modal Relation Extraction (MMRE) aims at identifying the relation between two entities in texts that contain visual clues. Rich visual content is valuable for the MMRE task, but existing works cannot well model finer associations among different modalities, failing to capture the truly helpful visual information and thus limiting relation extraction performance. In this paper, we propose a novel MMRE framework to better capture the deeper correlations of text, entity pair, and image/objects, so as to mine more helpful information for the task, termed as DGF-PT. We first propose a prompt-based autoregressive encoder, which builds the associations of intra-modal and inter-modal features related to the task, respectively by entity-oriented and object-oriented prefixes. To better integrate helpful visual information, we design a dual-gated fusion module to distinguish the importance of image/objects and further enrich text representations. In addition, a generative decoder is introduced with entity type restriction on relations, better filtering out candidates. Extensive experiments conducted on the benchmark dataset show that our approach achieves excellent performance compared to strong competitors, even in the few-shot situation.

## introduction as a fundamental subtask of information extraction, relation extraction (re) aims to identify the relation between two entities @xcite . recently, there is a growing trend in multi-modal relation extraction (mmre), aiming to classify textual relations of two entities as well as introduce the visual contents. it provides additional visual knowledge that incorporates multi-media information to support various cross-modal tasks such as the multi-modal knowledge graph construction @xcite and visual question answering systems @xcite . ## related work multi-modal relation extraction (mmre) task, a subtask of multi-modal information extraction in nlp @xcite @xcite , aims to identify textual relations between two entities in a sentence by introducing visual content @xcite , which compensates for insufficient semantics and helps to extract the relations. recently, there are several works @xcite beginning to focus on the multi-modal relation extraction technology. as the first work, mnre @xcite developed a multi-modal relation extraction baseline model. it demonstrates that introducing multimodal information supplements the missing semantics and improves relation extraction performance in social media texts. later, @xcite proposed a multi-modal neural network containing a scene graph modeling the visual relations of objects and aligning the relations between objects and text via similarity and attention mechanism. hvpnet @xcite designs a visual prefixguided fusion for introducing object-level visual information and further utilizes hierarchical multiscaled visual features. moreover, they introduce the information of all objects and thus cannot distinguish the truly helpful visual information, making it impractical to personalize the use of the image and further damaging their performance. in the multi-modal relation extraction task, we note that images are naturally helpful information in the problem of multi-modal relation extraction. however, the potential for a differentiated use of image information in this task is under-explored. in this paper, we focus on the finer (intra-modal and inter-modal) association and manage to integrate truly useful visual information, promoting the exploitation of limited images. this enables bridging the gap through the transfer multi-modal relation extraction task into mlm pre-train mechanism @xcite . ## problem formulation we provide the definition of mmre. ## framework this section introduces our proposed dgf-pt framework, as shown in figure 2 . we first design the prompt-based autoregressive encoder to acquire the fine-grained representations (entity pair, object, and text), which contains two types of prefixes for integrating helpful information and characterizing intra-modal and inter-modal interactions. to avoid unhelpful visual information misleading the model, we design the dual-gated fusion module to distinguish the importance of image/objects via local and global gates. it also integrates the semantics of image transferred by oscar @xcite . the fusion module outputs an enhanced representation. later, the generative decoder is proposed for relation prediction, leveraging the implicit associations and restricting candidate relations by introducing entity types. finally, we design the joint objective including distribution-consistency constraint, self-identification enhancement, and relation classification for model optimization. ## conclusion we propose dgf-pt, a novel multi-modal relation extraction framework, to capture deeper correlations among entity pair, text, and image/objects and integrate more helpful information for relation extraction. our framework effectively integrates intra-modal and inter-modal features, distinguishes helpful visual information, and restricts candidate relations. extensive experiments conducted on the benchmark dataset show that our approach achieves excellent performance.