Instructed Language Models with Retrievers Are Powerful Entity Linkers (2023)

Abstract: Generative approaches powered by large language models (LLMs) have demonstrated emergent abilities in tasks that require complex reasoning abilities. Yet the generative nature still makes the generated content suffer from hallucinations, thus unsuitable for entity-centric tasks like entity linking (EL) requiring precise entity predictions over a large knowledge base. We present Instructed Generative Entity Linker (INSGENEL), the first approach that enables casual language models to perform entity linking over knowledge bases. Several methods of equipping language models with EL ability were proposed in this work, including (i) a sequence-to-sequence training EL objective with instruction-tuning, (ii) a novel generative EL framework based on a light-weight potential mention retriever that frees the model from heavy and non-parallelizable decoding, achieving 4 \times speedup without compromise on linking metrics. INSGENEL outperforms previous generative alternatives with +6.8 F1 points gain on average, also with a huge advantage in training data efficiency and training compute consumption. In addition, our skillfully-engineered in-context learning (ICL) framework for EL still lags behind INSGENEL significantly, reaffirming that the EL task remains a persistent hurdle for general LLMs.

## introduction entity linking (el) has emerged as a critical research problem in the intersection of natural language processing (nlp) and information retrieval (ir), and it serves as a fundamental task in various nlp applications, including text understanding @xcite , question answering @xcite , to name a few. previous el approaches typically divided el into two steps: mention detection (md) and entity disambiguation (ed). once the md model produces mention span proposals over the input document, dense ed models attempt to encode mention context and entity candidates into representations. then a simple maximum inner product search (mips) is utilized to capture mention-entity correspondence, resulting in final entity predictions over the entire knowledge base (kb). a more recent trend in el research focuses on building an end-to-end pipeline that intertwines md and ed and formulates them into different tasks, such as question-answering @xcite , multi-task learning @xcite and language generation @xcite . while generative large language models (llms) have shown dominant abilities in a multitude of nlp tasks @xcite @xcite , their abilities are under-explored in the realm of entity-centric nlp tasks, especially el. unlike numerous knowledge language grounding tasks that can be readily unified to text-to-text frameworks @xcite , thanks to their close adherence to a unified surface form (i.e., text), el presents distinctive challenges. the foremost difficulty lies in the fact that unconstrained generation frequently fails to yield precise entity identifiers, because of the notorious hallucination issue of llms. even though the pre-defined prefix trie proposed in @xcite can ensure a legal generation sequence, the non-parallel beam search process makes it unsuitable for real-time usage, not to mention its performance lags behind later discriminative alternative. in this work, we revisit generative el by proposing three variants: ins-genel, insgenel-r and insgenel-icl. insgenel solves el by leveraging a methodology which constrains the next possible tokens, and eliminates invalid options during generation, thus ensuring the generated text can be successfully parsed into el results. in contrast to @xcite , the baseline model in ins-genel diverges by incorporating the input document as part of the prompt and optimizes the casual language modeling (clm) objective on decoder-only transformer models through instruction fine-tining approach, as opposed to employing a sequence-to-sequence objective based on an encoder-decoder neural architecture. we observe that by instruction fine-tuning decoder-only lms such as opt-series @xcite and llama-series @xcite , ins-genel yields superior el performance compared to previous work that finetunes a generative alternative on bart @xcite with +6.8 f1 points on average. this suggests that instruction fine-tuning may unlock specific entity-related knowledge within pretrained language models. additionally, this approach exhibits significant improvements in both training compute efficiency and data efficiency, indicating that foundation language models can effectively reduce the learning difficulty of task-specific objectives. however, directly generating sequences incurs significant computational overhead during inference, as both memory footprint and computation increase with sequence length, not to mention the non-parallelizable nature of auto-regressive decoding. to address these challenges, we propose offloading the mention detection (md) responsibility to an external retriever. for each document, the external retriever selects top-k entities that possibly exist in the document and constructs a possible mention set. then the surface form matching process dynamically determines the range where decisions are needed during the generation. finally, greedy decoding is employed only when a choice is necessary (within a decision range), i.e., whether to start a mention, end an entity identifier, or choose among dynamic candidates. named insgenel-r, this novel el generation framework offers several key advantages: a) compared to constrained beam search, insgenel-r significantly reduces the number of heavy generation forwards at the cost of a simple vector retrieval. b) it does not fall into beam failure, i.e., getting stuck with improbable mentions during the process of generating mention spans, thereby wasting valuable inference compute. c) it is less likely to miss obvious mentions, while traditional generative el tends to make mistakes when generating mention boundaries. insgenel-r achieves 4× reduction in terms of the number of lm calls, reduces runtime by a similar ratio at the minor cost of performance decline. moreover, we extend the usage of the same retriever in an in-context learning (icl) manner on llms such as gpt-3 (brown et al., 2020), gpt-3.5 and codex, named insgenel-icl. side-by-side comparative results indicate that while generic llms can correctly adhere to the format of exemplars by learning in context, they are unable to match the same accuracy exhibited by insgenel-r. in summary, this paper pushes the generative el paradigm to new limits, both in terms of evaluation metrics and runtime performance. the novel retrieval-augmented generative el emulates an agent interacting with the dynamic environment, and sheds light on real-time generative el without incurring a substantial metric drop. the new in-context learning paradigm for el also indicates that current llms can not support an optimal icl solution for el. we release our code and checkpoints at https://github.com/ mrzilinxiao/insgenentitylinking . while early el methods @xcite treat el as decomposed subtasks, such as mention detection (md) and entity disambiguation (ed), a more recent trend is considering el an end-to-end task. @xcite first propose to use a neural-based model to conduct md and ed jointly. @xcite transform the el into a bio tagging problem by training a token-classification model with an external entity classification head. @xcite formulate el into a question-answering (qa) problem and borrowed the popular retrieve-then-read pipeline in qa. @xcite leverage the type and description of entities and employs aggregation of discriminative scores to obtain the final result. for the generative el paradigm, cao et al. 2021b our methods fall within the generative el category as well, but they stand out in several distinctive features. first, putting the input document in the instruction enables el generation on decoder-only casual language models rather than using an encoder to capture the global context. this approach enables us to benefit from the advancements in recent foundation models. moreover, the mention detection offloading alleviates the burden of non-parallel generation. the practice of invoking the generative model only within the decision range drastically enhances computation efficiency, also presenting an intriguing parallel with interactive agents in knowledge base question answering (kbqa) @xcite and the principle of interactive nlp @xcite . instruction-tuning @xcite ) usually refers to finetuning language models with a collection of tasks that are formulated as plain-text instructions. recent llm pre-training @xcite show that emergent abilities @xcite exhibit when model size, data and training compute scale. specifically, one of such abilities is that large models can leverage natural language instructions to solve language problems in a zero-shot fashion. when instructions alone can not guide satisfactory generation, in-context learning (icl) can guide llm to learn from in-context exemplars to perform complex reasoning, such as solving mathematical problems @xcite . although instruction-tuning was originally proposed for zero-shot language tasks, we find that tuning models using document instructions also leads to improved prediction and disambiguation of uncommon entities in generative el. taking inspiration from icl, we have extended our approaches to encapsulate el with the icl paradigm, facilitating an equitable comparison between insgenel-r and general-purpose llms. retrieval-augmented language models are an emerging class of models in the field of nlp. they offer innovative solutions to knowledge-related tasks by combining the power of language mod-els with the ability to retrieve relevant information from a large corpus of external knowledge. most works augment the input context with retrieved external documents, thus the encoded representation or generated sequence will be conditioned on external knowledge. @xcite firstly train an end-to-end language encoder with a dense retrieval system. @xcite finetune a general sequence-to-sequence model with an external retriever. our insgenel-r utilizes a lightweight encoder for each document to retrieve the most relevant entities, then uses a generative agent to eliminate impossible solutions dynamically. for similar entity-related retrieval scenarios, févry et al. (2020) is the first to integrate entity supervision on language understanding tasks. @xcite constructs an entity memory bank and allows dynamic aggregation of entity representations, boosting the performance of entity-intensive questionanswering and generation tasks. however, none of them use retrieved entities to explicitly guide text generation. ## conclusion we present three variations of generative entity linking (el) solutions, where insgenel realizes an improvement of +3.0 f1 points on popular el test sets (+6.8 increase compared to previous generative methods). built upon this model, we propose a novel retrieval-augmented generative el framework insgenel-r that determines when to invoke the large language decoder on the fly. moreover, our pioneering insgenel-icl marks the inception of in-context learning in el, despite necessitating additional follow-up research to achieve competitive results.