Good Visual Guidance Make A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction (2022)

Abstract: Multimodal named entity recognition and relation extraction (MNER and MRE) is a fundamental and crucial branch in information extraction. However, existing approaches for MNER and MRE usually suffer from error sensitivity when irrelevant object images incorporated in texts. To deal with these issues, we propose a novel Hierarchical Visual Prefix fusion NeTwork (HVPNeT) for visual-enhanced entity and relation extraction, aiming to achieve more effective and robust performance. Specifically, we regard visual representation as pluggable visual prefix to guide the textual representation for error insensitive forecasting decision. We further propose a dynamic gated aggregation strategy to achieve hierarchical multi-scaled visual features as visual prefix for fusion. Extensive experiments on three benchmark datasets demonstrate the effectiveness of our method, and achieve state-of-the-art performance.

## introduction named entity recognition (ner) and relation extraction (re) are important tasks in information extraction and knowledge base population, due to its research significance in natural language processing (nlp) and wide applications @xcite @xcite . currently, with the rapid development of multimodal learning, multimodal ner (mner) and multimodal re (mre) methods @xcite have been proposed to enhance linguistic representations with the aid of visual clues from images. it significantly extends the text-based models by taking images as additional inputs, since the visual contexts help to resolve ambiguous multi-sense words. the madness outside of [lebron per] 1 house now includes a car crash. the essence of mner and mre tasks is how to learn great visual features and how to incorporate it into textual representation for enhancing ner and re. early methods @xcite study how to incorporate the feature of whole image into the textual representation. @xcite ; @xcite ; @xcite further validate that object-level visual fusion is more specific and important for mner and mre. recently, rpbert @xcite propose to train a classifier of whether the "image adds to the tweet meaning" before mner tasks. however, they heavily rely on pre-training on large extra annotated corpus of image-text relevance and only focus on the whole image with ignoring the bias of relevant object-level visual fusion. in practice, irrelevant objects may directly exert negative effects on the text inference. meanwhile, it is not trivial to acquire absolutely relevant object-level visual information to enhance the text. thus, an effective method should be derived to learn better visual representation and alleviate error sensitivity of irrelevant object images for social media ner and re tasks. considering images often appear before the text in a web document, we argue that images can be regarded as the prefix for their textual descriptions, which is inspired by prompt learning @xcite @xcite in the language model. specifically, given a image-text pair, we prepend object-level image feature sequence of length v i @xcite to the text sequence at each self-attention layer of bert @xcite . note that the visual prefix is a pluggable operation and don't require any annotation on relevance. therefore, visual prefix can not only introduce object-level visual signals, but also further reduce the impact on the architecture representing text. intuitively, visual prefix regarded as a prompt for text may helps alleviate the error sensitivity of irrelevant object images. while convolution neural networks (cnns) contain the multi-scale information with pyramidal feature hierarchy @xcite from low to high levels. and bert encodes a rich hierarchy of linguistic information @xcite from the bottom to the top. inspired by @xcite ; @xcite that objects of different sizes can have appropriate feature representations at the corresponding scales, we propose to make each layer of bert aware of hierarchical multiscale visual features to make a more enlightened and comprehensive forecasting decision. to this end, we propose a novel hierarchical visual prefix fusion network (hvpnet) for visual-enhanced entity and relation extraction. specifically, inspired by simvlm @xcite , we propose visual prefix-guided fusion mechanism involving concatenate object-level visual representation as the prefix of each selfattention layer in bert, which is a more soft and robust attention module for visual enhanced ner and re. we further design a dynamic gate for each layer to generate image-dependent paths, so that a variety of aggregated hierarchical multi-scaled visual features can be considered as visual prefix for enhancing ner and re. overall, we summerize the major contributions of our paper as follows: thus, textual representation of each layer in transformer can be aware of corresponding hierarchical visual features adaptively. to the best of our knowledge, this paper is the first work to leverage hierarchical pyramidal visual features for multimodal learning. our experimental results on three benchmark datasets validate the effectiveness and superiority of our hvpnet 2 related work multimodal entity and relation extraction as the crucial components of information extraction, named entity recognition (ner) and relation extraction (re) have attracted much attention in the research community @xcite @xcite . previous studies typically focus on textual modality and standard text. as multimodal data become increasingly popular on social media platforms, early research focusing on textual modality and standard text is limited. recently, several studies have focused on the mner and mre task, aiming to utilize the associate images to recognize the named entities and their relation better. in the early stages, @xcite , @xcite , @xcite and @xcite propose to encode the text through rnn and the whole image through cnn, then designing implicit interaction to model information between two modalities to explore multimodal ner tasks. recently, @xcite ; @xcite propose to leverage regional image features to represent objects in the image to exploit fine-grained semantic correspondences based on transformer and visual backbones. while most of the current methods ignore the error sensitivity, one exception is that @xcite , which proposes to learn a text-image relation classifier to enhance multimodal bert to reduce the interference from irrelevant images while requiring extensive annotation for the irrelevance of image-text pairs. ## methodology as illustrated in figure 2 , we present a novel hierarchical prefix fusion network for multi-modal entity and relation extraction. note that our method can also be applied to other visual-enhanced tasks towards text. ## conclusion and future work in this paper, we propose a novel hierarchical visual prefix fusion network (hvpnet) for visualenhanced entity and relation extraction. to be specific, we present visual prefix-guided fusion by concatenating object-level visual representation as the prefix of each self-attention layer in bert, which is a more soft and robust attention module for visual enhanced ner and re. we further design leveraging hierarchical multi-scaled visual representation as visual guidance for fusion. intuitively, good visual guidance make a better extractor, and extensive experimental and results on three benchmarks have demonstrated the effectiveness and robustness of our proposed method. meanwhile, our method also face the limitation that they don't suitable for mulimodal tasks in visual side, such as visual grounding. in the future, we plan to 1) explore more applications of hierarchical visual prefix in multimodal representation learning, making it more flexible and extensible; 2) try to apply the reverse version of our approach to boost visual representation with text for cv; 3) extend our approach to multitask multimodal pre-training.