Improving Candidate Generation for Low-resource Cross-lingual Entity Linking (2020)

Abstract: Cross-lingual entity linking (XEL) is the task of finding referents in a target-language knowledge base (KB) for mentions extracted from source-language texts. The first step of (X)EL is candidate generation, which retrieves a list of plausible candidate entities from the target-language KB for each mention. Approaches based on resources from Wikipedia have proven successful in the realm of relatively high-resource languages, but these do not extend well to low-resource languages with few, if any, Wikipedia pages. Recently, transfer learning methods have been shown to reduce the demand for resources in the low-resource languages by utilizing resources in closely related languages, but the performance still lags far behind their high-resource counterparts. In this paper, we first assess the problems faced by current entity candidate generation methods for low-resource XEL, then propose three improvements that (1) reduce the disconnect between entity mentions and KB entries, and (2) improve the robustness of the model to low-resource scenarios. The methods are simple, but effective: We experiment with our approach on seven XEL datasets and find that they yield an average gain of 16.9% in Top-30 gold candidate recall, compared with state-of-the-art baselines. Our improved model also yields an average gain of 7.9% in in-KB accuracy of end-to-end XEL.1

## introduction entity linking (el; @xcite @xcite associates entity mentions in a document with their entries in a knowledge base (kb). in this work, we focus on cross-lingual entity linking (xel; @xcite where the documents are in a source language that differs from the kb language (target). xel is an important component task for information extraction in languages that do not have extensive kb resources, and can potentially benefit downstream applications such as crosslingual building question answering systems @xcite , or supporting international humanitarian assistance efforts in areas that do not speak english @xcite . following @xcite and @xcite , we consider the target language kb to be english wikipedia. given a document and named entity mentions identified by a named entity recognition (ner) model, there are two primary steps in an xel system: (1) candidate generation, in which a model retrieves a short list of plausible kb entities for each mention and (2) disambiguation, in which a model selects the most likely kb entity from the candidate list. the quality of candidate lists will influence the performance of the end-to-end xel system, as correct entities not included in this list will not be recovered by the disambiguation model. in monolingual el, candidate generation has often been considered trivial @xcite . simple approaches using string similarity or wikipedia anchor-text links produce mention-entity lookup tables with high candidate recalls (e.g., in the 90% range), and thus most work focuses on methods for downstream entity disambiguation @xcite @xcite @xcite . string similarity (e.g., edit distance) cannot easily extend to xel because surface forms of entities often differ significantly across the source and target language, particularly when the languages are in different scripts. wikipedia link methods can be extended to xel by using inter-language links between the two languages to redirect entities to the english kb @xcite @xcite . this method works to some extent, but often under-performs on lowresource languages due to the lack of source language wikipedia resources. although scarce, there are some methods that propose to improve entity candidate generation by training translation models with low resourcelanguage (lrl)-english entity gazetteers @xcite , or learning neural string matching models based on an entity gazetteer in a related high-resource language (hrl) which is then applied to the lrl @xcite (more in §2). however, even with these relatively sophisticated methods, top-30 candidates still fall far behind their high-resource counterparts, lagging by as much as 70% absolute candidate recall. in this work, we perform a systematic study to understand and address the limitations of previous xel candidate generation models. first, in §3 we examine the sources of error in the state-of-the-art candidate generation model of @xcite , and identify a number of potential reasons for failure. specifically, we find that two common sources of error are (1) mismatch between the entity name in the kb and the entity mention in the text, and (2) failure of the string matching model itself. in figure 1 , we show an example of linking marathi, a low-resource language spoken in western india, to english, which we will use as a running example throughout the paper (although our method is broadly applicable, as noted in experiments). in this case, errors of the first type are due to the fact that the english entity cobie smulders is mentioned as (green, smulders) or (yellow, jacoba francisca maria smulders) in the text. errors of the second type are simple recognition errors such as where the mention (blue, cobie smulders) is recognized as english entity cobie sikkens. we proceed to propose methodological improvements that resolve these major issues. the first set of improvements handles the mismatch between the unique entity name that appears in the english kb, and the many different realizations of it in the source text. first, we note that training data used in learning-based methods for xel candidate generation @xcite is made of entity-entity pairs, which fail to capture this variation. we experiment with adding mention-entity pairs to the training data to provide explicit supervision, helping the model better capture the differences between mentions and entities ( §4.1). second, we note that many of the variations in the source language are actually similar to how the entity varies in english, and thus we can use english language resources to capture this variation. to this effect, we collect entity aliases from english wikidata foot_2 and allow the model to also look up these aliases during the candidate generation process ( §4.2). the second contribution of this work is a better modeling strategy for strings that represent mentions and entities ( §4.3). we posit that part of the reason why the lstm-based model of @xcite fails to properly model all words in a string is because it is not the ideal architecture to learn from limited training data, and as a result, it erroneously learns that some words in the mention can be ignored. to solve this problem, we replace the lstm with a more direct model based on the sum of character n-gram embeddings @xcite , which we posit is more likely to generalize to this difficult learning setting. we evaluate our proposed methods on four realworld xel datasets provided by darpa lorelei @xcite , as well as three other datasets we create with wikipedia anchortext and inter-language links ( §5). although our methods are simple, they are highly effective-our proposed model leads to gains ranging from 7.4-33.3% in top-30 gold candidate recall compared with @xcite in seven lrls. because our model provides downstream disambiguation models with a much larger headroom for improvement, we find that simply changing the candidate generation process yields an average gain of 7.9% in end-to-end xel in-kb accuracy in four lrls, pushing low-resource xel a step towards high-resource xel performance. ## failures of existing models in this section, we perform a systematic analysis of failure cases existing in pbel ( §3.1), and specifically focus on two error types: entity-mention mismatch ( §3.2) and string matching failures ( §3.3). ## improved candidate generation based on the results of this empirical study, we propose three methods to resolve the main problems inherent in the baseline pbel model. ## conclusion in this work, we perform a systematic analysis to study and address the limitation of a previous candidate generation model in low-resource settings. we propose three methodological improvements to resolve two main problems of the baseline model, namely, mismatch between mention and entity and sub-optimal string modeling. for the first problem, we introduce mention-entity pairs into the training process to provide supervision. we additionally collect entity aliases from english wikidata to further bridge this gap. to solve the second problem, we replace the lstm with a more direct model charagram. these methods form our proposed candidate generation model. we experiment with seven realistic datasets in lrls. our model yields an average gain of 16.9% in top-30 gold candidate recall. we also evaluate the influence of our candidate generation model in the context of end-to-end low-resource xel. it brings an average gain of 7.9% in four lrls. an immediate future focus is finding a way to properly combine multiple models trained on different hrls together to have better character n-gram coverage and thus improve model performance in different lrls. another interesting avenue is to investigate how to efficiently compare mentions and a large number of entities (e.g., 2m in wikipedia) in high dimensional space. currently, our model calculates the cosine similarity between a mention and every entity in the kb, which takes a few minutes for each test set. however, there is much existing work @xcite for efficient similarity search in high dimensional space for billion-scale datasets. it is likely that combination of these algorithms with our retrieval method will allow them to scale well and reduce the computation time to a few seconds. in addition, other interesting future directions are examining how to balance the trade-off between the gold candidate recall and the disambiguation difficulty, and how to apply our model to settings where the target language is not english.