Coherent Entity Disambiguation via Modeling Topic and Categorical Dependency (2023)

Abstract: Previous entity disambiguation (ED) methods adopt a discriminative paradigm, where prediction is made based on matching scores between mention context and candidate entities using length-limited encoders. However, these methods often struggle to capture explicit discourse-level dependencies, resulting in incoherent predictions at the abstract level (e.g. topic or category). We propose CoherentED, an ED system equipped with novel designs aimed at enhancing the coherence of entity predictions. Our method first introduces an unsupervised variational autoencoder (VAE) to extract latent topic vectors of context sentences. This approach not only allows the encoder to handle longer documents more effectively, conserves valuable input space, but also keeps a topic-level coherence. Additionally, we incorporate an external category memory, enabling the system to retrieve relevant categories for undecided mentions. By employing step-by-step entity decisions, this design facilitates the modeling of entity-entity interactions, thereby maintaining maximum coherence at the category level. We achieve new state-of-the-art results on popular ED benchmarks, with an average improvement of 1.3 F1 points. Our model demonstrates particularly outstanding performance on challenging long-text scenarios.

## introduction entity disambiguation (ed) is a typical knowledgeintensive task of resolving mentions in a document to their corresponding entities in a knowledge base (kb), e.g. wikipedia. this task is of great importance due to its active presence in downstream tasks such as information extraction @xcite , question answering @xcite and web search query @xcite . to perform efficient entity disambiguation (ed), one common approach is to encode mentions and candidate entities into different embedding spaces . then a simple vector dot product is used to capture the alignment between mentions and candidate entities. while this method enables quick maximum inner product search (mips) over all candidates and efficiently determines the linked answer, it suffers from late and simplistic interaction between mentions and entities @xcite . recently, researchers have proposed alternative paradigms for solving the ed problem, such as formulating it as a span extraction task @xcite . in this approach, a longformer @xcite is fine-tuned to predict the entity answer span within a long sequence consisting of the document and candidate entity identifiers. another paradigm @xcite reduces the ed task to an auto-regressive style in which generation models are trained to produce entity identifiers token-by-token. although these approaches offer some mitigation for the late-and-simple interaction problem, they still exhibit certain vulnerabilities. for instance, transformer-based encoders impose inherent limitations on input length, preventing the capture of long-range dependency for specific mentions. also, these methods do not explicitly consider coherence constraints, while coherence is considered as important as context in early ed works @xcite . we first propose to condition the model on compressed topic tokens, enabling the system to sustain topic coherence at the document level. in addition, the relationship among entities holds significant importance in the ed task. for example, mentions in the document exhibit a high correlation at the category level, where we name it category coherence. however, previous bi-encoder and cross-encoder solutions have overlooked these entity dependencies and focused solely on learning contextualized representations. among other works, extractive paradigm @xcite neglects entity-entity relation as well; generative el @xcite do possess some dependencies when linking an unknown mention. however, these dependencies arise from the auto-regressive decoding process and require heavy inference compute. to address the above coherence problem, we propose two orthogonal solutions that target topic coherence and entity coherence, respectively. following previous works that decode masked tokens to link unknown mentions @xcite , we present the overview of our coherent entity disambiguation work in figure 1 , where document words and unresolved entities are treated as input tokens of transformer @xcite . first, we bring an unsupervised variational auto-encoder (vae) @xcite to extract topic embeddings of surrounding sentences, which are later utilized to guide entity prediction. by docking these two representative language learners, bert @xcite and gpt-2 @xcite , the variational encoder can produce topic tokens of sentences without training on labeled datasets. this approach promotes a higher level of coherence in model predictions from an abstract level @xcite ) (e.g. tense, topic, sentiment). moreover, in most kbs, categories serve as valuable sources of knowledge, grouping entities based on similar subjects. to enhance entity-entity coherence from a categorical perspective, we design a novel category memory bank for intermediate entity representations to query dynamically. as opposed to retrieving from a frozen memory layer, we introduce direct supervision from ground-truth category labels during pre-training. this enables the memory to be learned effectively even from random initialization. named coherented, experimental results show that our proposed methods surpass previous state-of-the-art peers on six popular ed datasets by 1.3 f1 points on average. notably, on the challenging cweb dataset, which has an average document length of 1,700 words, our approach elevates the score from the previous neural-based sota of 78.9 to 81.1. through model ablations, we verify the effectiveness of the two orthogonal solutions through both quantitative performance evaluation and visualization analysis. these ablations further affirm the superiority of our methods in generating coherent disambiguation predictions. another intriguing feature for entities to differentiate from each other is their type, as entities with similar surface forms (text identifiers) often possess different types. onoe and durrett (2020) disambiguate similar entities solely through a refined entity type system derived from wikipedia, without using any external information. @xcite augment the robustness of such a type system even further. furthermore, many researchers have explored new paradigms for ed. @xcite propose using a prefix-constrained dictionary on casual language models to correctly generate entity strings. @xcite recast disambiguation as a task of machine reading comprehension (mrc), where the model selects a predicted entity based on the context fused with candidate identifiers and the document. the architecture of our system seamlessly blends the advantages of type systems and knowledge pretraining: incorporating type systems as dynamically updating neural blocks within the model, our design enables simultaneous learning through a multi-task learning schema -facilitating topic variation learning, masked disambiguation learning, and knowledge pre-training concurrently. prompt compression is a commonly used technique in language models to economize input space, closely related to topics such as prompt compression @xcite and context distillation @xcite . their mutual goal is to dynamically generate soft prompt tokens that replace original tokens without hurting downstream application performance. our topic token design mirrors context compression to some extent but differs in the compression ratio and purpose. our design is more compact; each context sentence gets transformer t [cls] t mess t ##i t played t in word / entity embedding t [sep] e lionel_messi e [mask] t the t world t cup c word c word c word c word c word c word c word c word c word c entity c entity p 0 p 1 p 2 p 3 p 4 p 5 p 6 p 7 p 8 e 1 + e 2 / 2 e 6 + e 7 / 2 [cls] mess ##i token type embedding position embedding played in the world cup [sep] lionel_messi [mask] document input entity input entity decoder prob entity vocab fifa_world_cup : tied weight : candidate mask figure 1: overview of our baseline model. legends are presented in the upper left corner. position embedding for an entity is averaged on corresponding document positions, for example, e 1 + e 2 / 2 indicates the entity "lionel_messi" is predicted based on document tokens mess and ##i at position 1 and 2. ## conclusion we propose a novel entity disambiguation method coherented, which injects latent topic vectors and utilizes cheap category knowledge sources to produce coherent disambiguation predictions. specifically, we introduce an unsupervised topic variational auto-encoder and an external category memory bank to mitigate inconsistent entity predictions. experimental results demonstrate the effectiveness of our proposed methods in terms of accuracy and coherence. analysis of randomly picked cases and vector visualizations further confirm such effectiveness.