Zero-shot Cross-Language Transfer of Monolingual Entity Linking Models (2022)

Abstract: Most entity linking systems, whether mono or multilingual, link mentions to a single English knowledge base. Few have considered linking non-English text to a non-English KB, and therefore, transferring an English entity linking model to both a new document and KB language. We consider the task of zero-shot cross-language transfer of entity linking systems to a new language and KB. We find that a system trained with multilingual representations does reasonably well, and propose improvements to system training that lead to improved recall in most datasets, often matching the in-language performance. We further conduct a detailed evaluation to elucidate the challenges of this setting.

## introduction entity linking -the process of matching mentions of people, places or organizations with a relevant knowledge base (kb) entry -has often focused on linking english text. cross-language linking often uses english kbs for matching to non-english text. while transferring a system to a new document language presents challenges, it does not consider issues that arise when transferring to a new kb language. kbs in different languages consider different topics, and matching text within the same language presents different challenges from building cross-language representations. people build kbs in many different languages, and we should explore how to link documents to these kbs. this paper considers zero-shot cross-language adaptation of a trained entity linking system to a new monolingual setting: the same new language for both the query document and kb. we consider adaptation so as to utilize the extensive annotated data resources for english, improving entity linking on languages that have little to no training data. consider the example in figure 1 , which links the spanish language mention senado (sen-ate) to the kb entry senado de la república (senate of the republic of mexico). an entity linker uses the mention text and surrounding sentence paired with the kb entry (including information such as the name, description) to score the likelihood of a match. many approaches to entity linking learn these linkages by training on a set of handannotated links in the desired language. if there are no or few language-specific annotations, how can we train a model on an annotation-rich language to perform well on other languages? similar to the architecture used in a crosslanguage setting @xcite , we take a neural approach to entity linking and use a multilingual pretrained transformer model, xlm-roberta (xlm-r) @xcite , to build representations of the available text for a mention and candidate entity pair. we feed each of these representations through a feed forward neural model to produce a likelihood score. xlm-r is a multilingual model that yields robust representations of text in a wide variety of languages. however, we find that even with the cross-language ability of xlm-r, in-language annotation data is key to an accurate linker. we thus propose ways to improve zero-shot cross-language transfer of a trained linker from one language to another. we adapt a method from @xcite to add an adversarial objective to linker training which uses an intermediate layer in the linker to transform language-specific embeddings to language-agnostic via a language classification module. similar approaches @xcite have been used in other multilingual nlp tasks, but have yet to be explored in el. to train this language-agnostic layer, we force the language classifier alone to predict the incorrect language label for unannotated portions of the source (e.g., english) and target (e.g., spanish) text. we jointly train the ranker and the language classifier using the correct source (e.g., english) language labels. ...lo acompañan el presidente del senado ... name senado de la república desc. el senado de los estados unidos ... which encourages the name and mention representation to be language-independent. second, we augment the entity linker with information from the target language kb to capture popularity of each entity, better handling entities that are common in the target language but rare in the source. we find that both model adjustments improve zero-shot performance on several language pairs, and that the adversarial model specifically produces consistent improvement in recall. overall, we demonstrate that entity linking models can be effectively adapted to a new language for both the query document and kb. ## entity linking model figure 1 shows an example mention in spanish (senado) linked to a spanish-language kb entry -senado de la república for the spanish mention. a linker will compare the text of the mention to the name of the entity, and consider information available in the context of the mention (the surrounding sentences). the entity description, and the mention and entity types. one approach to handling linking in multiple langages is to train separate models. while this works well for languages with a large amount of annotated data (english), others have far less (spanish). additionally, training a new model for each language does not scale well to many languages. instead, we pursue building a model that can be trained on entity linking annotations in a single language and transferred to another without additional annotations: cross-language entity linking. ## multilingual transfer the use of xlm-r makes our model inherently multilingual, allowing a single model to build representations in several languages. while this allows our models to do fairly well on previously unseen languages, we consider ways to further improve models during transfer: adaptation of the name matching model, and adaptation to the new knowledge base. ## model evaluation we begin with a zero-shot evaluation: how well does a model trained on english (tac) transfer to a new language without in-language training data? this baseline, which uses the same architecture as @xcite , leverages only the crosslingual ability of xlm-r to apply english language annotations to the new languages. we evaluate the english trained model on spanish (es) and chinese (zh) for tac, and russian (ru) and farsi (fa) for wiki. we also train a separate model for each of these languages to establish an in-language performance baseline. we illustrate the difference in performance of an english-only model as compared to an in-language trained one in figure 3 ; the dashed line above each metric shows the increase in performance. to control for the effect of training set size we ensure that the training sets are of equivalent size for each language by randomly downsizing the larger training dataset (e.g., english) to match the smaller (e.g., spanish). for comparison, we include a simple nearest neighbor baseline (noted as nn), which selects the highest scoring mention-entity pair using cosine similarity between the mention name m s and the entity representation e s . we then apply our language (noted as +a) and kb (noted as +ap) adaptation strategies for each language, and measure the performance on both the target and english language. in all cases, reported metrics are averaged over three runs. we report results for each language in the form of microaveraged precision (micro), recall (r), and f 1 . see appendix ## related work many studies on entity linking @xcite @xcite @xcite @xcite @xcite @xcite have served as the basis for developing cross-language systems, as has increasing research in monolingual model transfer in other information extraction tasks @xcite . one multilingual model is raiman and raiman (2018), which transfers an english-trained system to french-language wikipedia. they formulate a type system as a mixed integer problem, which they use to learn a type system from knowledge graph relations. their training approach uses broad amounts of annotated data with type information (e.g., all of english wikipedia). since we do not train english wikipedia models, and also do not use that magnitude of training data, we were not able to produce numbers using their system that are comparable to ours despite our best efforts to do so. other recent work @xcite uses a neural approach to link mentions in multiple languages, but differs from us by targeting language-agnostic kbs that include text in multiple languages. work using unsupervised graph methods, such as @xcite , are applied in non-english language pairs, such as chinese, but are not transferred from a secondary language. the related task of cross-language entity linking motivates approaches like transliteration @xcite , or monolingual entity linking paired with translation @xcite . some @xcite use the cross-language structure of wikipedia to build entity linkers, or @xcite study cross-language entity linking on low-resource languages. ## conclusion we explored how to build a monolingually-trained entity linker that can be transferred to new languages that do not have annotated training data. with a neural ranker model using xlm-r, we see that while in-language trained models perform better than english-trained models applied to second languages, the performance decrease is not large. we have validated several ways to improve these zero-shot models and find that an adversarial language classifier improves recall and f 1 on many datasets. furthermore, by adjusting the adversarial parameters, different performance objectives can be achieved, such as maximizing recall. we also present an analysis of our models, demonstrating which settings have the highest expectation of success. overall, we find that training the model to learn language-invariant representations is effective in improving performance when transferring to both text and a kb in a new language. a dataset tac the training set consists of mentions across 447 documents, and the evaluation set consists of mention annotations across 502 documents. this leaves us 14, 793 development mentions, of which 11, 344 are non-nil. wiki some basekb entities used in the tac dataset have wikipedia links provided; we used those links as seed entities for retrieving mentions, retrieving a sample mention of those and adding the remaining links in the page. we mark 20% of the mentions as nil. triage we use the system discussed in for both the tac and wiki datasets. however, while the triage system provides candidates in the same kb as the wiki data, not all entities in the tac kb have wikipedia page titles. therefore, the tac triage step requires an intermediate step -using the wikipedia titles generated by triage (@xmath0, we query a lucene database of basekb for relevant entities. for each title, we query basekb proportional to the prior provided by the triage system, meaning that we retrieve more basekb entities for titles that have a higher triage score, resulting in @xmath1. first, entities with wikipedia titles are queried, followed by the entity name itself. if none are found, we query the mention stringthis provides a small increase in triage recall.