Multilingual Autoregressive Entity Linking (2022)

Abstract: We present mGENRE, a sequence-to- sequence system for the Multilingual Entity Linking (MEL) problem—the task of resolving language-specific mentions to a multilingual Knowledge Base (KB). For a mention in a given language, mGENRE predicts the name of the target entity left-to-right, token-by-token in an autoregressive fashion. The autoregressive formulation allows us to effectively cross-encode mention string and entity names to capture more interactions than the standard dot product between mention and entity vectors. It also enables fast search within a large KB even for mentions that do not appear in mention tables and with no need for large-scale vector indices. While prior MEL works use a single representation for each entity, we match against entity names of as many languages as possible, which allows exploiting language connections between source input and target name. Moreover, in a zero-shot setting on languages with no training data at all, mGENRE treats the target language as a latent variable that is marginalized at prediction time. This leads to over 50% improvements in average accuracy. We show the efficacy of our approach through extensive evaluation including experiments on three popular MEL benchmarks where we establish new state-of-the-art results. Source code available at https://github.com/facebookresearch/GENRE .

## introduction entity linking (el, @xcite @xcite is an important task in nlp, with plenty of applications in multiple domains, spanning question answering @xcite @xcite , dialogue @xcite @xcite @xcite , and biomedical systems @xcite , to name just a few. it consists of grounding entity mentions in unstructured texts to kb descriptors (e.g., wikipedia articles). the multilingual version of the el problem has been for a long time tied to a purely cross-lingual formulation @xcite , where mentions expressed in one language are linked to a kb expressed in another (typically english). recently, @xcite made a step towards a more inherently multilingual formulation by defining a language-agnostic kb, obtained by grouping language-specific descriptors per entity. such a formulation has the power of considering entities that do not have an english descriptor (e.g., a wikipedia article in english) but have one in some other languages. a common design choice to most current solutions, regardless of the specific formulation, is to provide a unified entity representation, either by collating multilingual descriptors in a single vector or by defining a canonical language. for the common bi-encoder approach @xcite , this might be optimal. however, in the recently proposed genre model @xcite , an autoregressive formulation to the el problem leading to stronger performance and considerably smaller memory footprints than bi-encoder approaches on monolingual benchmarks, the representations to match against are entity names (i.e., strings) and it's unclear how to extend those beyond a monolingual setting. in this context, we find that maintaining as much language information as possible, hence figure 1 : mgenre: the input is text where an entity mention is signaled with special separator tokens and the model outputs predictions for the entity identifier. we use an autoregressive transformer decoder to generate language ids as well as entity names (i.e., wikipedia titles). the combination of language id and a entity name uniquely identify a wikidata id (with a n-to-1 mapping). we use beam search for efficient inference and we marginalize the probability scores for different languages to score entities. this example is a real output from our system and scores values are length normalized log-probabilities. concretely, in this paper, we present mgenre, the first multilingual el system that exploits a sequence-to-sequence architecture to generate entity names in more than 100 languages left to right, token-by-token in an autoregressive fashion and conditioned on the context (see figure 1 for an outline of our system). while prior works use a single representation for each entity, we maintain entity names for as many languages as possible, which allows exploiting language connections between source input and target name. to summarize, this work makes the following contributions: ## background we first introduce multilingual entity linking in section 2.1, highlighting its difference with monolingual and cross-lingual linking. we address the mel problem with a sequence-to-sequence model that generates textual entity identifiers (i.e., entity names). our formulation generalizes the genre model by de @xcite to a multilingual setting (mgenre). thus in section 2.2 and 2.3, we discuss the genre model and how it ranks entities with beam search respectively. ## model to extend genre to a multilingual setting, we need to define what are the unique identifiers of all entities in a language-agnostic fashion. this is not trivial because we rely on text representations that are by their nature grounded in some language. concretely, for each entity e, we have a set of identifiers i e that consists of pairs l, n l e where @xmath0. we extract these identifiers from our kb-each wikidata item has a set of wikipedia pages in multiple languages linked to it, and in any given language, each page has a unique name. we identify 3 strategies to employ these identifiers: i) define a canonical textual identifier for each entity such that there is a 1-to-1 mapping between the two (i.e., for each entity, select a specific language for its name-see section 3.1); all of these strategies define a different way we compute the underlining likelihood of our model. in figure 1 we show an outline of mgenre. the following subsections will present detailed discussions of the above 3 strategies. ## experimental setting we use wikidata @xcite as our kb while exploiting the supervision signal from wikipedia hyperlinks. for evaluation, we test our model on two established cross-lingual datasets, tr2016 hard and tac-kbp2015 @xcite , as well as the recently proposed mewsli-9 mel dataset @xcite . additionally, we propose a novel setting extracted from wikinews 2 where we train a model on a set of languages, and we test it on unseen ones. ## conclusion in this work, we propose an autoregressive formulation to the multilingual entity linking problem. for a mention in a given language, our solution generates entity names left-to-right and token-by-token. the resulting system maintains entity names in as many languages as possible to exploit language connections and interactions between source mention context and target entity name. the constrained beam search decoding strategy enables fast search within a large set of entity names (e.g., the whole kb in multiple languages) with no need for large-scale dense indices. we additionally design a novel objective that marginalizes over all available languages to perform a prediction. we show that this strategy is effective in dealing with languages for which no training data are available (i.e., 50% improvement for languages never seen during training). overall, our experiments show that mgenre achieves new state-of-the-art performance on three popular multilingual entity linking datasets. figure 6 : venn diagram of the overlap of languages used during multilingual language modeling (pre-training), the languages available on wikipedia (as of 2019-10-01), and the languages used by @xcite . after pre-training on 125 languages, we fine-tune on the 105 that overlap with the one available in wikipedia. search (i.e., the third alignment strategy), the mapping could be ambiguous (e.g., multiple items may share the same labels and aliases). we use a standard wikipedia extractor wikiextractor foot_5 by @xcite and a redirect extractor. 10 we use both wikipedia and wikidata dumps from 2019-10-01. a.3 data for test mewsli-9 @xcite ) contains 289,087 entity mentions appearing in 58,717 originally written news articles from wikinews, linked to wikidata. the corpus includes documents in 9 languages. 11 differently from the cross-lingual setting, this is a truly multilingual dataset since 11% target entities in mewsli-9 do not have an english wikipedia page. tr2016 hard @xcite is a wikipedia based cross-lingual dataset specifically constructed to contain difficult mention-entity pairs. authors extracted wikipedia hyperlinks for which the corresponding entity is not the most likely when using an alias we do not evaluate on mentions that link to entities out of the kb. previous work considered freebase @xcite as kb, and thus we computed a mapping between freebase id and wikidata id. when we cannot solve the match, our system results in a zero score (i.e., it counts as a wrong prediction). tac-kbp2015 contains 166 chinese documents (84 news and 82 discussion forum articles) and 167 spanish documents (84 news and 83 discussion forum articles) for a total of 12,853 mention-entity datapoints.