Trust in Internal or External Knowledge? Generative Multi-Modal Entity Linking with Knowledge Retriever (2024)

Abstract: Multi-modal entity linking (MEL) is a challenging task that requires accurate prediction of entities within extensive search spaces, utilizing multi-modal contexts. Existing generative approaches struggle with the knowledge gap between visual entity information and the intrinsic parametric knowledge of LLMs. To address this knowledge gap, we introduce a novel approach called GELR, which incorporates a knowledge retriever to enhance visual entity information by leveraging external sources. Additionally, we devise a prioritization scheme that effectively handles noisy retrieval results and manages conflicts arising from the integration of external and internal knowledge. Moreover, we propose a noise-aware instruction tuning technique during training to finely adjust the model’s ability to leverage retrieved information effectively. Through extensive experiments conducted on three benchmarks, our approach showcases remarkable improvements, ranging from 3.0% to 6.5%, across all evaluation metrics compared to strong baselines. These results demonstrate the effectiveness and superiority of our proposed method in tackling the complexities of multi-modal entity linking.

## introduction entity linking is a crucial component of information extraction that plays a fundamental role in various knowledge applications @xcite @xcite @xcite . it serves as a foundational tool to support tasks like visual question answering @xcite and semantic search by enabling the acquisition of entity-specific knowledge. over the years, there have been significant advancements in textual entity linking @xcite ., * the work was done when xinwei worked as intern at pattern recognition center, wechat ai, tencent inc, china. † corresponding author ... bruce banner from marvel's theavengers ... ... montcada be just a few kilometre awary ... context image context text generate w/o retrieval q13927 montcada i reixac municipality in the province of barcelone, spain.. q1010291montcada ,valencia comarca of horta nord in the valencian community q23781155 avengers: endgame, 2019 film directed by anthony and joe russot... q182218 the avengers (2012 film), the film's development began when marvel studios... retriever multi-modal llms generate w. noisy retrieval 2023b; @xcite , which involves establishing connections between mentions in unstructured text and entities in knowledge bases (kbs). however, the challenges arise when applying entity linking to social media contexts @xcite , where the understanding of multi-modal content becomes paramount. in these scenarios, the ability to link ambiguous mentions to their corresponding entities requires a comprehensive comprehension of multi-modal information, as illustrated in fig. 1 . consequently, the field of entity linking has extended its focus to encompass multi-modal scenarios. for mel, two main types of approaches are recognized: 1) retrieval-based methods @xcite excel at gathering potentially relevant contexts but often struggle to precisely identify the correct one. 2) generative approaches @xcite generate entities using the intrinsic knowledge present in large language models (llms). llms achieve better performance in top-ranked predictions, especially for common entities, thanks to their larger parameter size and stronger comprehension abilities. nevertheless, llms face challenges in associating unseen visual inputs with their internal entity knowledge. this gap makes it difficult to establish connections between fine-grained visual patterns and the corresponding entities. in such cases, combining retrieval-based and llm-based methods becomes beneficial. by obtaining relevant information through matching, the model can access its internal knowledge and make predictions, leveraging the strengths of both approaches. this integration enhances the overall performance of the entity-linking system. to this end, we introduce gelr, which bridges the knowledge gap between visual patterns and the parametric knowledge of llms by incorporating knowledge retrievers, achieving a balanced integration of extrinsic and intrinsic knowledge. however, utilizing external knowledge via retrievers can introduce additional issues when conflicts arise between parametric and retrieved knowledge @xcite . as depicted in fig. 1 , the linker adopts noisy retrieval results rather than relying on its own knowledge, leading to incorrect predictions "montcada i reixac". to discern noisy retrieval results and manage knowledge conflicts, we devise a prioritization scheme: gelr makes predictions conditioned on the retrieved entities if the retrieval results contain accurate clues; otherwise, the model relies exclusively on intrinsic parametric knowledge for predictions. towards this, we propose noise-aware instruction tuning to fine-tune the model in utilizing retrieved results by adjusting the predicted distribution for relevant and irrelevant knowledge, respectively. for entity generation during inference, our finetuned model directly utilizes the weighted difference between retrieval-aware and retrieval-free logits as the output logits. note that the relevance of retrieved entities is not given as prior information to the model, and our gelr solely determines its output based on the semantic features of multi-modal inputs and retrieved results. we carry out primary experiments on three benchmarks with knowledge bases ranging from 112k to 6m. the experimental results show significant improvements of 2.2-6.5% across all metrics compared to strong baselines. additionally, the experimental analysis highlights our method's effectiveness in leveraging both internal and external knowledge and alleviating the popularity bias in generative entity-linking methods. ## related work the traditional entity linking (el) task @xcite , which involves identifying mentions in text and linking them to corresponding entities in a knowledge base (kb), has been extensively researched. existing methods for el can be categorized into two types. the first type @xcite utilizes retrieval techniques to select the most relevant entities from an external knowledge base. the second type @xcite leverages the intrinsic knowledge within language models to directly generate entities associated with mentions. while textual entity linking has achieved significant advancements @xcite in accuracy on traditional benchmarks such as aida @xcite , challenges arise in social media applications @xcite , where the understanding of multi-modal contexts and entities is required. multi-modal entity linking (mel) is an extended version of the textual entity linking task, which focuses on linking textual or visual mentions with multi-modal contexts to their corresponding entities in the kb @xcite . to address these challenges, most methods adopt a two-stage retrieval framework @xcite @xcite , which involves recalling candidate entities and re-ranking the candidate set to select the best match. however, these straightforward methods heavily rely on high-quality candidate sets and suffer from cascading errors @xcite . moreover, existing retrievers excel in acquiring relevant contexts but struggle to precisely identify the correct one, leading to performance bottlenecks in entity-related tasks. recently, large language models (llms) have achieved remarkable success due to their impressive generative capabilities. notably, studies @xcite have emerged as an efficient finetuning paradigm where most parameters are frozen, and only newly added alignment modules are trained, surpassing the performance of the clip model. building on this success, subsequent research @xcite has extended the application of llms to other multi-modal tasks. in addition to efficient fine-tuning and leveraging the intrinsic knowledge of llms, there has been research @xcite focusing on retrieval-augmented multi-modal methods that incorporate external knowledge using retrievers. there have been preliminary attempts to use llms as virtual knowledge bases to address multimodal entity linking tasks @xcite @xcite . different from them, we leverage retrieved visual contexts as exter-nal potential knowledge to bridge the gap between different modalities, rather than simply using them as direct output. furthermore, we focus on learning the integration of both noisy external knowledge and intrinsic knowledge to make accurate predictions. ## methodology we propose gelr, a generative framework with knowledge retrievers for multi-modal entity linking, which learns to utilize knowledge from both external sources and intrinsic parameters of llms. gelr consists of three components, as depicted in fig. 2 : multi-modal knowledge retrieval, noiseaware instruction tuning, and inference. ## conclusion in this paper, we propose a generative framework with knowledge retrievers for multi-modal entity linking tasks. this framework leverages knowledge retrievers to bridge the gap between multi-modal input and parametric knowledge of llms. we carry out primary experiments on three benchmarks. the experimental results show significant improvements of 2.2% to 6.5% across all metrics compared to strong baselines.