Granular Entity Mapper: Advancing Fine-grained Multimodal Named Entity Recognition and Grounding (2024)

Abstract: Multimodal Named Entity Recognition and Grounding (MNERG) aims to extract paired textual and visual entities from texts and images. It has been well explored through a two-step paradigm: initially identifying potential visual entities using object detection methods and then aligning the extracted textual entities with their corresponding visual entities. However, when it comes to fine-grained MNERG, the long-tailed distribution of textual entity categories and the performance of object detectors limit the effectiveness of traditional methods. Specifically, more detailed classification leads to many low-frequency categories, and existing object detection methods often fail to pinpoint subtle regions within images. To address these challenges, we propose the Granular Entity Mapper (GEM) framework. Firstly, we design a multi-granularity entity recognition module, followed by a reranking module based on the Multimodal Large Language Model (MLLM) to incorporate hierarchical information of entity categories, visual cues, and external textual resources collectively for accurate fine-grained textual entity recognition. Then, we utilize a pre-trained Large Visual Language Model (LVLM) as an implicit visual entity grounder that directly deduces relevant visual entity regions from the entire image without the need for bounding box training. Experimental results on the GMNER and FMNERG datasets demonstrate that our GEM framework achieves state-of-the-art results on the fine-grained content extraction task.

## introduction multimodal named entity recognition and grounding (mnerg) aims to recognize named entities and corresponding image regions from multimodal data, which is crucial for various applications, including multimodal knowledge graph construction, video recommendation, and multimodal chatbot. typical mnerg approaches often involve a two-step framework @xcite , where a well-trained object detection model is utilized to extract image regions as potential visual entities. then, a cross-modality modeling framework is leveraged to extract and link textual entities with corresponding potential visual entities, enabling multimodal entity alignment. along this line, numerous efforts have been recently dedicated to exploring this problem, and notable performances have been achieved. moreover, to better capture the complexity of the real world, fine-grained mnerg endeavors to classify textual entities into more detailed categories and extract smaller, more precise visual entity regions. indeed, delving into fine-grained mnerg reveals new challenges and limitations. on the one hand, fine-grained textual entities often suffer from the problem of long-tailed distribution, necessitating external information sources to achieve precise recognition and classification of these textual entities. on the other hand, finegrained visual entities often exhibit a wide variety of sizes, which challenges traditional object detection methods in consistently recalling them and further hinders multimodal entity alignment. for example, as shown in figure 1 , the textual entity one direction requires common knowledge about the band and the individuals in the image to help discriminate it from other organization categories. additionally, existing object detection methods can only detect coarse-grained potential visual entity regions in the figure, and the logo as the corresponding visual entity does not appear among the candidates due to its small size. therefore, supplementing the valuable knowledge and clues and tracing relevant regions directly from the images is essential for fine-grained content extraction. fortunately, recent years have witnessed the prosperity of multimodal large models @xcite @xcite , which have shown advanced capabilities in comprehending relationships and reasoning in complex scenarios involving texts and images. inspired by such progress, we fully utilize the cross-modal interacting capabilities of various multimodal large models and propose a novel fine-grained mnerg framework, named granular entity mapper (gem), to address the above challenges. firstly, we employ a knowledge-enhanced multigranular entity recognition module, followed by a multimodal reranking module, to incorporate external textual knowledge, structured information, and visual cues collectively for accurate finegrained textual entity recognition. specifically, we acquire rich external knowledge from large language models (llms) through prompts and then preliminarily recognize entities constrained by the entity category hierarchy to enhance long-tailed categories. leveraging the powerful relationship comprehension and endogenous multimodal knowledge of multimodal large language models (mllms foot_0 ), we rerank the predicted textual entity categories to differentiate long-tailed categories from similar categories. secondly, we utilize a large visual language model (lvlm) as an implicit grounder to establish associations between textual entities and their corresponding visual entity regions, enabling the recognition of visual entities even without training on annotated bounding boxes. due to the numerous natural text and image alignments during the pre-training stage, our grounder is suitable for open-vocabulary textual entities and can directly identify the corresponding regions across the image, overcoming the limitations associated with traditional object detectors for fine-grained visual entity grounding. the main contributions of our work can be sum-marized as follows: ## method in this section, we first formulate the fine-grained mnerg task and then explain our framework in detail. our gem comprises three main modules: ## discussion in this section, we detail our preference for using the mllm with instruction-tuning for reranking instead of a larger model with in-context learning. furthermore, our results show that the blip outperforms existing mllms in visual entity grounding. more discussions are in appendix d, e, f. ## conclusion in this paper, we introduced gem, a novel framework for fine-grained multimodal named entity recognition and grounding based on integrated multi-granularity and multi-level information. by harnessing the rich multimodal knowledge and linguistic understanding from multimodal pretraining, we enhanced the comprehension of finegrained information in both images and texts. extensive experimental results demonstrated the superior performance of the gem framework.