LNN - EL : A Neuro-Symbolic Approach to Short-text Entity Linking (2021)

Abstract: Entity linking (EL) is the task of disambiguating mentions appearing in text by linking them to entities in a knowledge graph, a crucial task for text understanding, question answering or conversational systems. In the special case of short-text EL, which poses additional challenges due to limited context, prior approaches have reached good performance by employing heuristics-based methods or purely neural approaches. Here, we take a different, neuro-symbolic approach that combines the advantages of using interpretable rules based on first-order logic with the performance of neural learning. Even though constrained to use rules, we show that we reach competitive or better performance with SoTA black-box neural approaches. Furthermore, our framework has the benefits of extensibility and transferability. We show that we can easily blend existing rule templates given by a human expert, with multiple types of features (priors, BERT encodings, box embeddings, etc), and even with scores resulting from previous EL methods, thus improving on such methods. As an example of improvement, on the LC-QuAD-1.0 dataset, we show more than 3% increase in F1 score relative to previous SoTA. Finally, we show that the inductive bias offered by using logic results in a set of learned rules that transfers from one dataset to another, sometimes without finetuning, while still having high accuracy.

## introduction entity linking (el) is the task of disambiguating textual mentions by linking them to canonical entities provided by a knowledge graph (kg) such as dbpedia, @xcite or wikidata @xcite . a large body of existing work deals with el in the context of longer text (i.e., comprising of multiple sentences) @xcite . the general approach is: 1) extract features measuring some degree of similarity between the textual mention and any one of several candidate entities @xcite @xcite , followed by 2) the disambiguation step, either heuristics-based (non-learning) @xcite @xcite or learning-based @xcite @xcite @xcite , to link the mention to an actual entity. a particular type of entity linking, focused on short text (i.e., a single sentence or question), has attracted recent attention due to its relevance for downstream applications such as question answering (e.g., @xcite ) and conversational systems. short-text el is particularly challenging because the limited context surrounding mentions results in greater ambiguity @xcite . to address this challenge, one needs to exploit as many features from as many sources of evidence as possible. consider the question in figure 1 (a), containing mention 1 (cameron) and mention 2 (titanic). 1 dbpedia contains several person entities whose last name matches cameron. two such entities are shown in figure 3 (b), james_cameron and roderick_cameron, along with their string similarity scores (in this case, character-level jaccard similarity) to mention 1 . in this case, the string similarities are quite close. in the absence of reliable discerning information, one can employ a prior such as using the more popular candidate entity, as measured by the in-degree of the entity in the kg (see figure 3(b) ). given the higher in-degree, we can (correctly) link mention 1 to james_cameron. however, for mention 2 , the correct entry is @xcite as opposed to titanic the ship, but it actually has a lower string similarity. to link to the correct entity, one needs to exploit the fact that james_cameron has an edge connecting it to @xcite in the kg (see ego network on the left in figure 1(c )). linking co-occurring mentions from text to connected entities in the kg is an instance of collective entity linking. this example provides some intuition as to how priors, local features (string similarity) and collective entity linking can be exploited to overcome the limited context in short-text el. while the use of priors, local features and nonlocal features (for collective linking) has been proposed before @xcite , our goal in this paper is to provide an extensible framework that can combine any number of such features and more, including contextual embeddings such as bert encodings @xcite and query2box embeddings @xcite , and even the results of previously developed neural el models (e.g., blink @xcite ). additionally, such a framework must not only allow for easy inclusion of new sources of evidence but also for interpretability of the resulting model @xcite ). an approach that combines disparate features should, at the very least, be able to state, post-training, which features are detrimental and which features aid el performance and under what conditions, in order to enable actionable insights in the next iteration of model improvement. ## lnn-el an overview of our neuro-symbolic approach for entity linking is depicted in figure 3 . we next discuss the details about feature generation component that generates features using a catalogue ## evaluation we first evaluate our approach w.r.t performance & extensibility, interpretability and transferability. we also discuss the training and inference time. datasets. as shown in baselines. we compare our approach to ( 1 ) blink (wu et al., 2020), the current state-of-theart on both short-text and long-text el, (2) three bert-based models -(a) bert: both mention and candidate entity embeddings are obtained via bert base pre-trained encoder, similar to (gillick et al., 2019), (b) bertwiki: mention embeddings are obtained from bert base , while candidate entity is from pretrained wiki2vec (yamada et al., 2020), (c) box: bertwiki embeddings finetuned with query2box embeddings (see section 4.1). in addition to the aforementioned black-box neural models, we also compare our approach to (3) two logistic regression models that use the same feature set as lnn-el: logisticregression without blink and logisticregression blin k with blink. furthermore, we use the following variants of our approach: (4) ruleel: a baseline rule-based el approach with manually defined weights and thresholds, (5) logicel: a baseline approach built on ruleel where only the thresholds are learnable, based on product t-norm (see section 3.2), (6) lnn-el: our core lnn-based method using non-embedding features plus spacy, and (7) lnn-el ens : an ensemble combining core lnn-el with additional features from existing el approaches, namely blink and box (we consider box, as it outperforms bert and bertwiki on all datasets). detailed rule templates are provided in appendix a.3. ## conclusions we introduced lnn-el, a neuro-symbolic approach for entity linking on short text. our approach complements human-given rule templates through neural learning and achieves competitive performance against sota black-box neural models, while exhibiting interpretability and transferability without requiring a large amount of labeled data. while lnn-el provides an extensible framework where one can easily add and test new features in existing rule templates, currently this is done manually. a future direction is to automatically learn the rules with the optimal combinations of features.