CHOLAN : A Modular Approach for Neural Entity Linking on W ikipedia and W ikidata (2021)

Abstract: In this paper, we propose CHOLAN, a modular approach to target end-to-end entity linking (EL) over knowledge bases. CHOLAN consists of a pipeline of two transformer-based models integrated sequentially to accomplish the EL task. The first transformer model identifies surface forms (entity mentions) in a given text. For each mention, a second transformer model is employed to classify the target entity among a predefined candidates list. The latter transformer is fed by an enriched context captured from the sentence (i.e. local context), and entity description gained from Wikipedia. Such external contexts have not been used in state of the art EL approaches. Our empirical study was conducted on two well-known knowledge bases (i.e., Wikidata and Wikipedia). The empirical results suggest that CHOLAN outperforms state-of-the-art approaches on standard datasets such as CoNLL-AIDA, MSNBC, AQUAINT, ACE2004, and T-REx.

## introduction the explicit schema, graph-based structure, and interlinking nature of information represented in publicly available knowledge graphs (kgs) e.g., dbpedia @xcite , freebase @xcite , wikidata @xcite or knowledge bases (kbs) such as wikipedia; introduce a new landscape of features, as well as structured knowledge and embeddings. researchers have developed several techniques to align information available in unstructured text to the concepts of these kgs @xcite . end-to-end entity linking (hereafter el) task follows this direction; such that, given a sentence el first identifies the entity mention in the sentence, then maps these mentions to the most likely kg/kb entities. the el comprises of a threestep process. with respect to the given example sentence soccer: late goals give japan win over syria, the first step called mention detection (md) identifies the surface forms japan and syria. the next step is candidate generation (cg) aiming to find a list of possible entity candidates in the kg/kb for each entity mention. for example, the candidates list for entity mention japan consists in part of japan national football team, japan (country), japan (band) and for syria is syria (roman province), syria national football team, greater syria. finally, the third step deals with the entity disambiguation (ed) which employs the coreference and contextual features to discriminate the most likely entity from the candidates list e.g., japan national football team and syria national football team are correct entities. entity linking approaches are broadly categorised into three categories. the initial attempts @xcite solve md and ed as independent sub-tasks of el (i.e., a pipeline based system). however, these approaches exhibit a behaviour where errors propagate from md to ed hence might downgrade the overall performance of the system. the second category has emerged in an attempt to mitigate these errors, where researchers focused on jointly modelling md and ed, emphasising the importance of the mutual dependency of the two sub-tasks @xcite . these two el approaches depend on an intermediate candidate generation step and rely on a pre-computed list of entity candidates. for example, @xcite propose a joint md and ed model and inherits the candidate list from @xcite . the third approach combines the three sub-steps in a joint model and illustrates that each of those tasks is interdependent @xcite . the recent el approaches focus on jointly modelling two or three subtasks @xcite . furthermore, the nlp research community has extensively used transformers in end-to-end models for entity linking @xcite @xcite ). nevertheless, these works report less performance than @xcite , which is a bi-lstm based model. the observations regarding the limited performance of transformer-based models for the el motivate our work, and in this paper, our focus is to understand the bottlenecks in the entity linking process. we argue that the less studied task in literature, i.e., candidate generation, has an essential role in the el models' performance, which has not been a focus in the recently proposed transformer-based entity linking models. in this paper, we hypothesise that the transformer models, though trained on a large corpus, may require additional task-specific contexts. furthermore, inducing the context at the entity disambiguation step may positively impact the overall performance, which has not been utilised in the state of the art methods due to monolithic implementations @xcite @xcite . subsequently, we deviate from the joint modelling of two or three subtasks of the el and revert to the methodology opted by earlier el systems in 2011 @xcite , i.e. treat each subtask independently. as such, we study the research question: rq: what is the impact of each sub-task (aka component) on the overall outcome of the transformer-based entity linking approach? we propose an intuitive novel approach named cholan, comprising a modular architecture of two transformer models to solve md and ed independently. in the first step, cholan employs bert @xcite model to identify mentions of the entities in an input sentence. the second step involves expanding each mention with a list of kb entity candidates. finally, the en-tity mention, sentence (local context), an entity candidate, and entity wikipedia description (entity context) are fed as input sequences in the second bert based model to predict the correct kb entity (cf. figure 1 ). we train md and ed steps independently during training, and while testing, we run the cholan pipeline end-to-end for predicting the kb entity. the following are the novel features of cholan: the implementation is publicly available foot_0 . the paper is structured as follows: next section summarises the related work. section 3 describes the problem statement and approach. section 4 explains the experimental settings followed by results in 5. we conclude in section 6. recently, contextual embedding based models achieve state of the art for ner/md task @xcite . we point to the survey by @xcite for details about ner. few early el models have performed md task independently @xcite . ## problem statement and approach we formally define el task as follows: the next task is candidate generation where for each mention m x a set of candidates c(m x )= {e x 1 , ..., e x n |e x @xmath0. finally, the entity disambiguation (ed) task aims to map each mention m @xmath1. in our case, we model the ed task as a classification task and augment the input with extra signals as context. for every candidate entity c @xmath2, the model estimates a probability p i , thus the most likely entity is the one with the highest probability as γ = arg max p i {p(p i | m x , c x i , w, c)} where w and c are the input representations respectively for the given sentence (local context) and the context derived from kg/kb. as such the probability of score p i is conditioned not only on m x and c x i but also on w and c as contextual parameters. ## conclusions in the last two years, the nlp research community has extensively tried transformer-based models for the el task. however, the performance remained lower than @xcite . this paper combines the traditional software engineering principle of modular architecture with the contextinduced transformers to effectively solve the el task. our reason to deviate from an end-to-end architecture was to provide full flexibility to our system in terms of candidate generation list, underlying kg, and induction of the context at the ed step. we attribute cholan's outperformance to the following reasons: 1) the modular architecture, which brings flexibility and interoperability as cholan can treat each task independently. @xcite reports that shifting towards joint modelling of md and ed tasks helps mitigate error propagation from md to ed. however, the performance of bert base for the md task is significantly high (92.3 on aida-b and 94.3 f1-score on t-rex calculated by us) remarkably reducing the errors in md. cholan leverages this capability in the md subtask, placing more focus on cg and ed tasks.