E ntity L inking in 100 L anguages (2020)

Abstract: We propose a new formulation for multilingual entity linking, where language-specific mentions resolve to a language-agnostic Knowledge Base. We train a dual encoder in this new setting, building on prior work with improved feature representation, negative mining, and an auxiliary entity-pairing task, to obtain a single entity retrieval model that covers 100+ languages and 20 million entities. The model outperforms state-of-the-art results from a far more limited cross-lingual linking task. Rare entities and low-resource languages pose challenges at this large-scale, so we advocate for an increased focus on zero- and few-shot evaluation. To this end, we provide Mewsli-9, a large new multilingual dataset matched to our setting, and show how frequency-based analysis provided key insights for our model and training enhancements.

## introduction entity linking (el) fulfils a key role in grounded language understanding: given an ungrounded entity mention in text, the task is to identify the entity's corresponding entry in a knowledge base (kb). in particular, el provides grounding for applications like question answering @xcite (also via semantic parsing @xcite ) and text generation @xcite ; it is also an essential component in knowledge base population @xcite . entities have played a growing role in representation learning. for example, entity mention masking led to greatly improved fact retention in large language models @xcite . but to date, the primary formulation of el outside of the standard monolingual setting has been cross-lingual: link mentions expressed in one language to a kb expressed in another @xcite @xcite . the accompanying motivation is that kbs may only ever exist in some well-resourced languages, but that text in many different languages need to be linked. recent work in this direction features progress on low-resource languages @xcite , zero-shot transfer @xcite @xcite and scaling to many languages @xcite , but commonly assumes a single primary kb language and a limited kb, typically english wikipedia. we contend that this popular formulation limits the scope of el in ways that are artificial and inequitable. first, it artificially simplifies the task by restricting the set of viable entities and reducing the variety of mention ambiguities. limiting the focus to entities that have english wikipedia pages understates the real-world diversity of entities. even within the wikipedia ecosystem, many entities only have pages in languages other than english. these are often associated with locales that are already underrepresented on the global stage. by ignoring these entities and their mentions, most current modeling and evaluation work tend to side-step underappreciated challenges faced in practical industrial applications, which often involve kbs much larger than english wikipedia, with a much more significant zero-or few-shot inference problem. second, it entrenches an english bias in el research that is out of step with the encouraging shift toward inherently multilingual approaches in natural language processing, enabled by advances in representation learning @xcite @xcite . third, much recent el work has focused on models that rerank entity candidates retrieved by an alias table @xcite , an approach that works well for english entities with many linked mentions, but less so for the long tail of entities and languages. to overcome these shortcomings, this work makes the following key contributions: ## task definition multilingual entity linking (mel) is the task of linking an entity mention m in some context language l c to the corresponding entity @xmath0. that is, while the kb may include textual information (names, descriptions, etc.) about each entity in one or more languages, we make no prior assumption about the relationship between these kb languages l @xmath1, . . . , l k } and the mention-side language: l c may or may not be in l kb . this is a generalization of cross-lingual el (xel), which is concerned with the case where l @xmath2. commonly, l is english, and v is moreover limited to the set of entities that express features in l . ## mewsli-9 dataset we facilitate evaluation on the proposed multilingual el task by releasing a matching dataset that covers a diverse set of languages and entities. mewsli-9 (multilingual entities in news, linked) contains 289,087 entity mentions appearing in 58,717 originally written news articles from wikinews, linked to wikidata. 4 the corpus includes documents in nine languages, representing five language families and entities lang. wikinews articles constitute a somewhat different text genre from our wikipedia training data: the articles do not begin with a formulaic entity description, for example, and anchor link conventions are likely different. we treat the full dataset as a test set, avoiding any fine-tuning or hyperparameter tuning, thus allowing us to evaluate our model's robustness to domain drift. ## model prior work showed that a dual encoder architecture can encode entities and contextual mentions in a dense vector space to facilitate efficient entity retrieval via nearest-neighbors search @xcite . we take the same approach. the dual encoder maps a mention-entity pair (m, e) to a score: our encoders are bert-based transformer networks @xcite , which we initialize from a pretrained multilingual bert checkpoint. 7 for efficiency, we only use the first 4 layers, which results in a negligible drop in performance relative to the full 12-layer stack. the wordpiece vocabulary contains 119,547 symbols covering the top 104 wikipedia languages by frequency-this is the language set we use in our experiments. ## experiments we conduct a series of experiments to gain insight into the behavior of the dual encoder retrieval models under the proposed mel setting, asking: ## conclusion we have proposed a new formulation for multilingual entity linking that seeks to expand the scope of entity linking to better reflect the real-world challenges of rare entities and/or low resource languages. operationalized through wikipedia and wikidata, our experiments using enhanced dual encoder retrieval models and frequency-based evaluation provide compelling evidence that it is feasible to perform this task with a single model covering over a 100 languages. our automatically extracted mewsli-9 dataset serves as a starting point for evaluating entity linking beyond the entrenched english benchmarks and under the expanded multilingual setting. future work could investigate the use of non-expert human raters to improve the dataset quality further. in pursuit of improved entity representations, future work could explore the joint use of complementary multi-language descriptions per entity, methods to update representations in a light-weight fashion when descriptions change, and incorporate relational information stored in the kb.