Information Screening whilst Exploiting! Multimodal Relation Extraction with Feature Denoising and Multimodal Topic Modeling (2023)

Abstract: Existing research on multimodal relation extraction (MRE) faces two co-existing challenges, internal-information over-utilization and external-information under-exploitation. To combat that, we propose a novel framework that simultaneously implements the idea of internal-information screening and external-information exploiting. First, we represent the fine-grained semantic structures of the input image and text with the visual and textual scene graphs, which are further fused into a unified cross-modal graph (CMG). Based on CMG, we perform structure refinement with the guidance of the graph information bottleneck principle, actively denoising the less-informative features. Next, we perform topic modeling over the input image and text, incorporating latent multimodal topic features to enrich the contexts. On the benchmark MRE dataset, our system outperforms the current best model significantly. With further in-depth analyses, we reveal the great potential of our method for the MRE task.

## introduction relation extraction (re), determining the semantic relation between a pair of subject and object entities in a given text @xcite , has played a vital role in many downstream natural language processing (nlp) applications, e.g., knowledge graph construction @xcite , question answering @xcite . but in realistic scenarios (i.e., social media), data is often in various forms and modalities (i.e., texts, images), rather than pure texts. thus, multimodal relation extraction has been introduced recently @xcite , where additional visual sources are added to the textual re as an enhancement to the relation inference. the essence of a successful mre lies in the effective utilization of multimodal information. certain efforts have been made in existing mre work and achieved promising performances, where delicate interaction and fusion mechanisms are designed for encoding the multimodal features @xcite . nevertheless, current methods still fail to sufficiently harness the feature sources from two information perspectives, which may hinder further task development. internal-information over-utilization. on the one hand, most existing mre methods progressively incorporate full-scale textual and visual sources into the learning, under the assumption that all the input information certainly contributes to the task. in fact, prior textual re research extensively shows that only parts of the texts are useful to the relation inference @xcite , and accordingly propose to prune over input sentences @xcite . the case is more severe for the visual inputs, as not all and always the visual sources play positive roles, especially on the social media data. as revealed by @xcite , as high as 33.8% of visual information serves no context or even noise in mre. @xcite thus propose to selectively remove im-ages from the input image-text pairs. unfortunately, such coarse-grained instance-level filtering largely hurts the utility of visual features. we argue that a fine-grained feature screening over both the internal image and text features is needed. taking the example #1 in fig. 1 , the textual expressions 'congratulations to angela and mark salmons' and the visual objects of 'gift' and 'roses' are valid clues to infer the 'couple' relation between ' angela' and 'mark salmons', while the rest of text and visual information is essentially the task-irrelevant noise. external-information under-exploitation. on the other hand, although compensating the text inputs with visual sources, there can be still information deficiency in mre, in particular when the visual features serve less (or even negative) utility. this is especially the case for social media data, where the contents are less-informative due to the short text lengths and low-relevant images @xcite . for the example #2 in fig. 1 , due to the lack of necessary contextual information, it is tricky to infer the relation 'present in' between 'hot summer' (an album name) and 'migos' (a singer name) based on both the image and text sources. in this regard, more external information should be considered and exploited for mre. fortunately, the topic modeling technique offers a promising solution, which has been shown to enrich the semantics of the raw data, and thus facilitate nlp applications broadly @xcite . for the above same example, if an additional 'music' topic feature is leveraged into the context, the relation inference can be greatly eased. taking into account the above two observations, in this work, we propose a novel framework to improve mre. as shown in fig. 4 , we first employ the scene graphs (sgs) @xcite to represent the input vision and text, where sgs advance in intrinsically depicting the fine-grained semantic structures of texts or images. we fuse both the visual and textual sgs into a cross-modal graph (cmg) as our backbone structure. next, we reach the first goal of internal-information screening by adjusting the cmg structure via the graph information bottleneck (gib) principle @xcite , i.e., gib-guided feature refinement, during which the less-informative features are filtered and the task-relevant structures will be highlighted. then, to realize the second goal of external-information exploiting, we perform multimodal topic integration. we devise a latent multimodal topic module to produce both the textual and visual topic features based on the multimodal inputs. the multimodal topic keywords are integrated into the cmg to enrich the overall contexts, based on which we conduct the final reasoning of relation for input. we perform experiments on the benchmark mre dataset @xcite , where the results show that our framework significantly boosts the current state of the art. further analyses demonstrate that the gib-guided feature refinement helps in effective input denoising, and the latent multimodal topic module induces rich taskmeaningful visual&textual topic features as extended contexts. we finally reveal that the idea of internal-information screening is especially important to the scenario of higher text-vision relevance, while the external-information exploiting particularly works for the lower text-vision relevance case. to sum up, this work contributes by introducing a novel idea of simultaneous information subtraction and addition for multimodal relation extraction. the internal-information over-utilization and external-information under-exploitation are two common co-existing issues in many multimodal applications, to which our method can be broadly applied without much effort. ## mre framework as shown in fig. 4 , our overall framework consists of five tiers. first, the model takes as input an image i and text t , as well as the subject v s and object entity v o . we represent i and t with the corresponding vsg and tsg. then, the vsg and tsg are assembled as a cross-modal graph, which is further modeled via a graph encoder. next, we perform gib-guided feature refinement over the cmg for internal-information screening, which results in a structurally compact backbone graph. afterwards, the multimodal topic features induced from the latent multimodal topic model are integrated into the previously obtained feature representation for external-information exploitation. finally, the decoder predicts the relation label y based on the enriched features. ## related works as one of the key subtasks of the information extraction track, relation extraction (re) has attracted much research attention @xcite @xcite . the recent trend of re has shifted from the traditional textual re to the recent multimodal re, where the latter additionally adds the image inputs in the former one for better performances, under the intuition that the visual information can offer complementary features to the purely textual input from other modalities. @xcite pioneers the mre task with a benchmark dataset, which is collected from the social media posts that come with rich vision-language sources. later, more delicate and sophisticated methods are proposed to enhance the interactions between the input texts and images, and achieve promising results @xcite ; @xcite . on the other hand, increasing attention has been paid to exploring the role of different information in the re task. as extensively revealed in prior re studies, only a few parts of the input sentence can provide real clues for the relation inference @xcite , which inspires the proposal of textual feature pruning methods @xcite . more recently, @xcite ; @xcite have shown that not always the visual inputs serve positive contributions in existing mre models, as the social media data contains many noises. @xcite thus introduce an instance-level filtering approach to directly drop out those images less-informative to the task. however, such coarse-grained aggressive data deletion will inevitably abandon certain useful visual features. in this work we propose screening the noisy information from both the visual and textual input features, in a fine-grained and more controllable manner, i.e., structure denoising via graph information bottleneck technique @xcite . also, we adopt the scene graph structures to model both the vision and language features, which partially inherits the success from @xcite that uses visual scene graphs to represent input images. due to the sparse and noisy characteristics of social media data, as well as the cross-modal information detachment, mre also suffers from feature deficiency problems. we thus propose modeling the latent topic information as additional context features to enrich the inputs. multimodal topic modeling has received considerable explorations @xcite , which extends the triumph of the textual latent topic models as in nlp applications @xcite @xcite . we however note that existing state-of-the-art latent multimodal models @xcite fail to navigate the text and image into a unified feature space, which leads to irrelevant vision-text topic induction. we thus propose an effective latent multimodal model that learns coherent topics across two modalities. to our knowledge, we are the first to attempt to integrate the multimodal topic features for mre. ## conclusion in this paper, we solve the internal-information over-utilization issue and the external-information under-exploitation issue in multimodal relation extraction. we first represent the input images and texts with the visual and textual scene graph structures, and fuse them into the cross-modal graphs. we then perform structure refinement with the guidance of the graph information bottleneck principle. next, we induce latent multimodal topic features to enrich the feature contexts. our overall system achieves huge improvement over the existing best model on the benchmark data. further in-depth analyses offer a deep understanding of how our method advances the task.