LLM s as Bridges: Reformulating Grounded Multimodal Named Entity Recognition (2024)

Abstract: Grounded Multimodal Named Entity Recognition (GMNER) is a nascent multimodal task that aims to identify named entities, entity types and their corresponding visual regions. GMNER task exhibits two challenging properties: 1) The weak correlation between image-text pairs in social media results in a significant portion of named entities being ungroundable. 2) There exists a distinction between coarse-grained referring expressions commonly used in similar tasks (e.g., phrase localization, referring expression comprehension) and fine-grained named entities. In this paper, we propose RiVEG, a unified framework that reformulates GMNER into a joint MNER-VE-VG task by leveraging large language models (LLMs) as a connecting bridge. This reformulation brings two benefits: 1) It maintains the optimal MNER performance and eliminates the need for employing object detection methods to pre-extract regional features, thereby naturally addressing two major limitations of existing GMNER methods. 2) The introduction of entity expansion expression and Visual Entailment (VE) module unifies Visual Grounding (VG) and Entity Grounding (EG). It enables RiVEG to effortlessly inherit the Visual Entailment and Visual Grounding capabilities of any current or prospective multimodal pretraining models. Extensive experiments demonstrate that RiVEG outperforms state-of-the-art methods on the existing GMNER dataset and achieves absolute leads of 10.65%, 6.21%, and 8.83% in all three subtasks.

## introduction multimodal named entity recognition (mner) @xcite aims to extract named entities and corresponding categories from image-text pairs sourced from social media. as one of the most * corresponding authors. figure 1 : pipeline-1 illustrates the challenges posed by gmner task to existing visual grounding methods. riveg builds a bridge between noun phrases and named entities and introduces visual entailment to address the weak correlation between image and text. analyzing gmner from the two perspectives of mner and eg, existing gmner methods @xcite have several significant limitations. limitation 1. entity grounding (eg) is an important part in gmner, which aims to determine the groundability and visual grounding results of text named entities predicted by the mner model. obviously, effective eg depends on accurate mner. but the mner performance of the existing multimodal fusion gmner methods @xcite is suboptimal. this may be due to the introduction of visual object features destroying the original text feature representation in mner @xcite . a series of mner studies in the text-text paradigm (i.e., represent images using text) @xcite show that text representation plays the most critical role in mner. models with the best mner performance typically do not involve any cross-modal interactions. however, existing end-to-end gmner methods @xcite violate this in order to obtain eg capabilities with the help of cross-modal interactions. this results in the feature representation learned by the model being neither the best for mner nor the best for eg. therefore, splitting the prediction and grounding of named entities into separate stages is necessary. limitation 2. existing gm-ner methods @xcite typically employ object detection (od) techniques to extract features from candidate regions, which are then aligned with text entities. but as shown in visual grounding (vg) @xcite aims to correlate natural language descriptions with specific regions in images. although vg has a similar paradigm to the entity grounding (eg) in gmner, existing vg methods @xcite @xcite exhibit significant deficiencies when applied to the gmner dataset. this is attributed to two fundamental distinctions between these two tasks: differentiation 1. referring to the pipeline-1 and pipeline-2 in figure 1 , the text input for the eg task consists of real-world named entities, which is notably different from the referring expressions or noun phrases typically used in the vg method. there is a substantial difference that is almost nongeneralizable between them. the vg method can given the above differences, the potential of vg methods cannot be unleashed in eg. considering that most vg methods do not require preextraction of regional features, we ask: is it possible to reformulate gmner as a two-stage union of mner-eg? the first stage focuses on optimizing the mner effect to solve the limitation 1 of the existing gmner method. the second stage uses the vg method to naturally bypass regional feature extraction to solve limitation 2. in the second stage, to adapt the vg method to the eg task, is it conceivable to establish a bridge between named entities and noun phrases to address differentiation 1 between the vg method and eg task? moreover, is it feasible to resolve differentiation 2 by devising a sensible filtering mechanism? in this paper, we introduce gmner into a multistage framework consisting of mner, @xcite . this modular design inherently mitigates two limitations of existing gmner methods, enabling the connection of three unrelated independent tasks in series within gmner and unlocking significant potential. specifically, in order to ensure the optimal performance of the mner stage, we leverage auxiliary refined knowledge distilled from large language models (llms) to aid in the identification of named entities. and we introduce a visual entailment (ve) module to handle the weak correlation between image and text. named entities are heuristically converted by llms into corresponding entity expansion expressions to facilitate the ve and vg modules to determine the groundability and grounding results of named entities. main contributions are summarized as follows: ## methodology riveg is mainly divided into two stages. in the stage of named entity recognition and expansion, we employ auxiliary refined knowledge obtained from llms to ensure optimal mner performance. furthermore, in order to adapt existing vg methods to eg task, llms is considered as a bridge between named entities and referring expressions. riveg instructs llms to convert named entities into named entity referring expressions (detailed in §3.2). in the stage of named entity grounding, riveg reformulates the entire eg task as a union of ve and vg. to reconcile the challenge between the weak correlation of image-text pairs in gmner and the strong predefined constraints of existing vg methods, we design the ve module for buffering. images and named entity referring expressions serve as the input to ve module for determining the groundability of the named entities. finally, the named entities deemed groundable alongside images are employed as inputs to vg module to obtain the final grounding results (detailed in §3.3). we also design a data augmentation method suitable for riveg. an overview of the riveg is depicted in figure 2 . ## conclusion in this paper, we propose riveg, a unified framework that aims to use llms as bridges to reformulate the gmner task into a unified mner-ve-vg task. this reformulation enables the framework to seamlessly incorporate the capabilities of stateof-the-art methods for each subtask in their corresponding modules. extensive experiments show that riveg significantly outperforms state-of-theart methods. the proposed riveg greatly reduces the constraints of vg and ve methods on natural language inputs, expands their applicability to more realistic scenarios, and provides insights for future work to better deploy vg and ve modules in related tasks.