Named Entity and Relation Extraction with Multi-Modal Retrieval (2022)

Abstract: Multi-modal named entity recognition (NER) and relation extraction (RE) aim to leverage relevant image information to improve the performance of NER and RE. Most existing efforts largely focused on directly extracting potentially useful information from images (such as pixel-level features, identified objects, and associated captions).However, such extraction processes may not be knowledge aware, resulting in information that may not be highly relevant.In this paper, we propose a novel Multi-modal Retrieval based framework (MoRe).MoRe contains a text retrieval module and an image-based retrieval module, which retrieve related knowledge of the input text and image in the knowledge corpus respectively.Next, the retrieval results are sent to the textual and visual models respectively for predictions.Finally, a Mixture of Experts (MoE) module combines the predictions from the two models to make the final decision.Our experiments show that both our textual model and visual model can achieve state-of-the-art performance on four multi-modal NER datasets and one multi-modal RE dataset.With MoE, the model performance can be further improved and our analysis demonstrates the benefits of integrating both textual and visual cues for such tasks.

## introduction utilizing images to improve the performance of named entity recognition (ner) and relation extraction (re) has attracted increasing attentions in natural language processing. image information can be utilized in various domains such as social media @xcite @xcite , movie reviews @xcite and news @xcite . most of the previous approaches utilize images by extracting information such as feature representations @xcite , object tags @xcite and image captions @xcite to improve the model performance. however, most of image feature, object tag and caption extractors are trained based on datasets such as imagenet @xcite and visual genome @xcite , which mainly contain common nouns foot_0 instead of named entities. as a result, the extractors (especially those of object tags and image captions) often output information about common nouns, which may not be helpful for entity-based task models. recently, pretrained vision-language models @xcite @xcite have improved the performance of crossmodal tasks such as vqa @xcite , nlvr @xcite and image-text retrieval @xcite significantly. however, suffering from the same problem, the pretrained visionlanguage models do not achieve better performance than textual models in multi-modal ner @xcite . recently, approaches based on text retrieval have shown their effectiveness over question answering @xcite @xcite , machine translation @xcite @xcite , language modeling @xcite , ner @xcite @xcite and entity linking @xcite . the approaches use the input texts as the search query to retrieve the related knowledge in the knowledge corpus (kc), which is a key-value structured memory built from the knowledge source. besides, humans can recognize the entities (such as famous persons and locations) in the image based on their learned knowledge in practice. when they are not sure about the entities in the image, they can even use image-based retrieval in the search engine to get the related knowledge about the image. inspired by that, for multi-modal ner and re models, we believe retrieving the related knowledge of the image can be utilized to help the task models to disambiguate the named entities as well. in this paper, we propose multimodal retrieval based framework (more), which explores the knowledge behind the input image and text pairs for multi-modal ner and re. more retrieves related knowledge for the input text and image using the textual retriever and image retriever respectively. the text retriever retrieves the most related paragraphs in the kc and the image retriever finds the documents containing the most related images. the retrieval results of each modality are sent to the textual and visual models respectively and used for training on ner and re tasks. after both of the models are trained, the mixture of experts (moe) module is trained to learn how to combine the model predictions from the two models. the contributions of more can be summarized in four aspects: ## more given an input text and image pair (x, i), where @xmath0, • • • , x n }, more aims to predict the outputs y. y can be the label sequence or the relations for the ner and re task respectively. more feeds (x, i) into the multi-modal retrieval module, which returns retrieved texts z t and z i from a text retrieval module and a image-based retrieval module respectively. the textual task model and visual task model concatenate the input sentence x with the retrieved knowledge z t and z i from kc and predict the output distribution p (y|x, i, z t ) and p (y|x, i, z i ) respectively. finally, the moe module fuses the predictions from the models based on each modality and makes the final prediction p (y|x, i). the architecture of our framework is shown in figure 1 . ## conclusion in this paper, we introduce a novel multi-modal retrieval based framework that utilizes the knowledge behind the multi-modal inputs. more first retrieves related knowledge of input text and image from a text retrieval module and an image-based retrieval module. more then feeds the retrieved knowledge from the text retrieval module and the image-based retrieval module into the textual and visual task models respectively to make predictions. given the predictions from the task models of each modality, more combines the prediction by a mixture of experts (moe) module. the moe module takes the features of each input text and image into consideration and makes the final decision. in our experiments, we show that both our textual model and visual model can achieve state-of-the-art performance on four multi-modal ner datasets and one multi-modal re dataset. with moe, the model performance can be further improved. in analysis, we demonstrate the advantage of integrating both textual and visual cues for such tasks over different types of labels.