Low-Rank Subspaces for Unsupervised Entity Linking (2021)

Abstract: Entity linking is an important problem with many applications. Most previous solutions were designed for settings where annotated training data is available, which is, however, not the case in numerous domains. We propose a light-weight and scalable entity linking method, Eigenthemes, that relies solely on the availability of entity names and a referent knowledge base. Eigenthemes exploits the fact that the entities that are truly mentioned in a document (the “gold entities”) tend to form a semantically dense subset of the set of all candidate entities in the document. Geometrically speaking, when representing entities as vectors via some given embedding, the gold entities tend to lie in a low-rank subspace of the full embedding space. Eigenthemes identifies this subspace using the singular value decomposition and scores candidate entities according to their proximity to the subspace. On the empirical front, we introduce multiple strong baselines that compare favorably to (and sometimes even outperform) the existing state of the art. Extensive experiments on benchmark datasets from a variety of real-world domains showcase the effectiveness of our approach.

## introduction entity linking (el) is the task of grounding mentions to a reference knowledge base (also referred to as knowledge graph). with a plethora of applications, including but not limited to information extraction @xcite and automatic knowledge base construction @xcite , el is one of the most actively researched topics in natural language processing. despite the recent proliferation of el methods, recent works @xcite have pointed out that the performance of existing techniques largely relies on the existence of large corpora of annotated data. unsupervised entity linking. in contrast, truly unsupervised entity linkers should be able to operate in the absolute absence of annotated data, with access to only a reference knowledge graph (kg) and a list of entity names, or aliases. furthermore, neither the candidate generator nor the disambiguation technique (the two key modules of any typical entity linker) can make use of annotated data. therefore, "unsupervised" disambiguation techniques that leverage labeled data to generate candidate entities @xcite are not applicable in our setting. only very recently have researchers @xcite started to focus on el systems that can operate in the absolute absence of annotated data. the motivation is well founded: there are some domains that link to their own specialized kgs, for which labeled data is not readily available, expensive to obtain, or scarce in the best case. this can be exemplified with domains such as law, science, or medicine. moreover, companies may have their proprietary kgs where some entities are only meaningful with respect to the company. in all these cases, available labeled data cannot generalize to the corresponding specialized kgs. challenges. the majority of the existing methods @xcite @xcite @xcite in the el literature are ill-suited for unsupervised el as they rely heavily on annotated data. these methods typically leverage annotated data to (1) generate candidate entities, (2) use features (e.g. the prior probability p (e|m) of an entity e given a mention m) derived from these annotations, (3) learn aligned word and entity embeddings enabling computation of similarity between an entity and the mention context, and (4) use a set of mentions and their corresponding gold entities to train supervised models. thus, it is non-trivial and unclear how to adapt these methods to work in the absence of annotated data. by the same token, one may not expect a large validation set of annotated data wherein to perform extensive hyperparameter tuning. thus, novel approaches to el without annotated data should either possess a small number of hyperparameters or be robust to hyperparameter tuning. ## problem and notation we first formalize the task of interest. let d be a single document from a collection d of documents. similar to most previous works in the el literature, with few exceptions @xcite , we assume that mention spans (usually obtained by a named entity recognizer) are provided. also, let m @xmath0, m 2 , . . . , m n } be the set of n mentions contained in d, and e be the set of entities in the reference kg g. a low-dimensional representation (embedding) can be learned for each entity by applying node representation learning techniques such as deepwalk @xcite to the graph g. the entity embeddings learned by these techniques are known to be meaningful with respect to the relatedness of the entities they represent @xcite . the el task consists in finding, for each mention @xmath1, the entity @xmath2. a key component of any el system is the candidate generation system. let c t denote such a system, which, for a given mention m, retrieves at most t entities-typically referred to as candidate entities. in the simplest case, the candidate generation system may retrieve all possible candidates (then @xmath3, but it is common to most of the works to retrieve a small subset of the most likely entities for that mention based on a certain sorting criterion. there is no consensus in the literature regarding a suitable value of t , and previous works have used varying values such as 7 @xcite , 20 @xcite or 30 @xcite . for each mention @xmath4, the set of candidate entities is given by c t (m) = {e 1 m , e 2 m , . . . , e t m }, where e i m indicates the i th candidate entity for the mention m. for the example introduced in § 1, the candidate entities for the mentions michael jordan and science are portrayed in fig. 1 . a second key component of any entity linker is the disambiguation module. this module selects an entity contained in the referent kg g for each mention of a document. the disambiguation process may be performed independently for each mention, or collectively for all mentions in a document. in the latter case, the fundamental assumption of all collective disambiguation algorithms is that the set of gold entities are topically related. setting. contrary to most of the existing work, where both the candidate generation and the disambiguation modules are dependent on annotated data, we focus on the setting where we only know the referent kg g and the mentions to be disambiguated, but have no annotated data that could generalize to the entities contained in g. ## entity linking with eigenthemes eigenthemes learns a low-rank subspace from the ensemble of embeddings corresponding to all candidate entities for all mentions within a document ( § 4.1). we will see in § 4.2 that the learning of the subspace may also be guided by different weights associated with the embeddings. the learned subspace is represented by what we refer to as "eigenthemes". the eigenthemes are components that are learned so as to decompose the ensemble of the entity embeddings as a linear combination of these components. the vector-subspace similarity function ( § 4.3) takes the learned low-rank subspace and the vector representation of a candidate entity, and computes a similarity score as a weighted sum of similarities between the entity vector representation and each of the eigenthemes of the subspace. the similarity between the entity representation and an eigentheme indicates how much of the signal from the former can be projected into the latter, and it is weighted with a value that relates to the strength of the eigentheme in the entity embedding ensemble of the document. as a result, only those entities that lie in the learned subspace or in its proximity will have a high score. it is important to note that while the vector-subspace similarity function is applied to each mention's candidates independently, the learned subspace encompasses information from all mentions in a document. therefore, the disambiguation performed by eigenthemes is collective, as the subspace given by the eigenthemes is learned at a global level in the document. additional details (pseudocode, architectural overview, etc.) about eigenthemes are presented in appendix b. ## experiments all the resources (code, datasets, etc.) required to reproduce the experiments in this paper are available at https://github.com/epfl-dlab/eigenthemes . ## analysis in this section, we perform a post-mortem analysis on a plethora of aspects impacting the downstream el performance of the considered methods measured using precision@1. results for mrr show similar trends and are therefore omitted. do learned subspaces capture the relatedness among gold entities? fig. 2 shows that gold entities tend to form tight clusters when projected on the first two components of the subspace learned by eigen. this result provides a data-driven justification regarding the existence of relatedness among gold entities mentioned within a documenta key assumption used in the design of almost every method in the el literature. moreover, it also showcases the ability of the subspaces learned by eigen to capture such relatedness. do gold entities lie closer to the learned subspaces when compared to other candidate entities? the disambiguation module of eigen relies on the gold entities of a document being closer to the learned subspace than other candidate entities. while the strong el performance of eigen portrayed in does the distribution of mention-types affect el performance? inspired by the cloze test (taylor, 1953) used to assess language learning capability of individuals @xcite , we perform a mutilation analysis to assess the impact of the distribution of mention-types on the downstream el performance. the test is carried out by removing varying number of 'easy' mentions from the conll-test dataset, thereby increasing the fraction of 'hard' (ambiguous) mentions, and thus, can be deemed analogous to adding noise in the dataset. specifically, we generate 11 different dataset versions by subsampling (uniformly at random) varying fraction (between 1 and 0 in decrements of 0.1) of easy mentions while retaining all the hard mentions, and measure the 'overall' el performance in each version. we run this experiment 10 times and report the mean performance. being very small, the standard deviations were omitted to avoid clutter. fig. 3a portrays that the overall el performance of all the techniques deteriorates with the reduction in the fraction of easy mentions, which is natural. interestingly, the deterioration observed for the degree baseline is much more profoundprecision@1 plummets from 0.57 (fraction-@xmath6, and it gets demoted from being the second best technique to the worst. note that even avg starts outperforming degree when fraction of easy mentions are less than 0.5. on the contrary, eigen consistently outperforms all other techniques in all the subsampled versions. this experiment establishes two important points. (1) it shows that the performance of de-gree is deceptively high. more importantly, it unveils a key limitation of degree, i.e., its inability to address challenging scenarios of el where the fraction of easy mentions is low. (2) it bolsters the superiority of eigen over other techniques by showcasing its ability to reliably and consistently perform well, even in challenging scenarios. robustness: out-of-domain datasets. it is standard practice in the el literature @xcite to evaluate the robustness of the models trained on the conlltrain dataset in out-of-domain datasets without any (re)validation of hyperparameters. in the same vein, the experiments on all the other datasets were performed using the hyperparameters tuned on the conll-val dataset. specifically, we assess the robustness of the techniques on the recently introduced wned-wiki and wned-clueweb datasets, and the wikilinks-random dataset of english wikipedia tables. the results are presented in fig. 3b . note that τ mil-nd cannot perform el in the wikilinks-random dataset owing to the absence of sequential textual content, and therefore, the corresponding bar in the plot is non-existent. it is worth noting that τ mil-nd (the existing sota) performs the worst in the out-of-domain scenario, whereas all other techniques exhibit robustness. hence, this experiment simply unveils a limitation of τ mil-nd, i.e., its lack of robustness, which is a noteworthy finding. consequently, the experiment also establishes two important properties of eigen: (1) robustness to hyperparameter tuning, and (2) adaptability to newer domains. ## conclusions in this paper, we addressed the problem of el in the absence of annotated data with a light-weight method, eigenthemes, that relies solely on the availability of entity names and a referent kb. experiments on benchmark datasets portrayed the effectiveness of our proposed approach. in the future, our aim is to validate the effectiveness of our approach in performing multi-lingual entity linking with a special focus on low-resource languages.