W ikipedia Entities as Rendezvous across Languages: Grounding Multilingual Language Models by Predicting W ikipedia Hyperlinks (2021)

Abstract: Masked language models have quickly become the de facto standard when processing text. Recently, several approaches have been proposed to further enrich word representations with external knowledge sources such as knowledge graphs. However, these models are devised and evaluated in a monolingual setting only. In this work, we propose a language-independent entity prediction task as an intermediate training procedure to ground word representations on entity semantics and bridge the gap across different languages by means of a shared vocabulary of entities. We show that our approach effectively injects new lexical-semantic knowledge into neural models, improving their performance on different semantic tasks in the zero-shot crosslingual setting. As an additional advantage, our intermediate training does not require any supplementary input, allowing our models to be applied to new datasets right away. In our experiments, we use Wikipedia articles in up to 100 languages and already observe consistent gains compared to strong baselines when predicting entities using only the English Wikipedia. Further adding extra languages lead to improvements in most tasks up to a certain point, but overall we found it non-trivial to scale improvements in model transferability by training on ever increasing amounts of Wikipedia languages.

## introduction pretrained multilingual masked language models (mmlms) such as mbert @xcite , xlm-r @xcite and their variants have achieved state-of-the-art results across diverse natural language understanding tasks. typically, a mmlm model is pretrained on very large amounts of raw text in different languages using the masked language modelling (mlm) objective and is further finetuned on (usually limited amounts of) task data. in the zero-shot crosslingual setting, which is our focus in this paper, a mmlm is finetuned on the target task using data in a single language (e.g., english) and is evaluated on the same task but in different languages (e.g., non-english languages). we introduce the multilingual wikipedia hyperlink prediction objective to contextualise words in a text with entities and concepts from an external knowledge source by using wikipedia articles in up to 100 languages. hyperlink prediction is a knowledge-rich task designed to (1) inject semantic knowledge from wikipedia entities and concepts into the mmlm token representations, and (2) with a similar motivation as the translated language modelling loss of @xcite , i.e., to inject explicit language-independent knowledge into a model trained via self-supervised learning, but in our case without parallel data. we devise a training procedure where we mask out hyperlinks in wikipedia articles and train the mmlm to predict the hyperlink identifier similarly to standard mlm but using a "hyperlink vocabulary" of 250k concepts shared across languages. we use the state-of-the-art mmlm xlm-rlarge @xcite and show that by adding an add-on training step using wikipedia hyperlink prediction we consistently improve several zero-shot crosslingual natural language understanding tasks across a diverse array of languages: crosslingual word sense disambiguation in 18 languages including english (xl-wsd; @xcite ; the crosslingual word-in-context task (xl-wic; @xcite in 12 non-english languages; and in 7 tasks from the xtreme benchmark @xcite in up to 40 languages. recently, @xcite and @xcite devised different methods to incorporate entities from external knowledge graphs into masked language model (lm) training. since then, several works followed @xcite @xcite 2020; yamada t a s k s mle target task -en target task en -loss t a s k s mle target task -en target task en -loss bn:00021494n computer science (en), informatica (it), 计算机科学 (zh), ... bn:00002705n [cls] ho anni studiato [ent] per diversi . [sep] [cls] ho studiato [mask] [mask] per diversi anni . [sep] [cls] [mask] [mask] is the study of [mask] . [sep] mmlm [cls] [ent] [ent] is the of . [sep] algorithm (en), algoritmo (it), 算法 (zh), … wikipedia hyperlink prediction training target task training target task evaluation t a s k s t a s k s mmlm target task -en target task en -loss ar l a n g u a g e s hidden states words [ent] [ent] study task 1 t a s k s mmlm task -ar task -ar target task -ar task 1 l a n g u a g e s t a s k s target task ar -prediction target task ar -prediction target task ar -prediction figure 1: we finetune a pretrained mmlm using multilingual wikipedia hyperlink prediction, then further train a model on a set of target tasks in english and evaluate on non-english data (i.e., zero-shot crosslingual setting). et al., 2020) showing increasingly better performance than masked lms that rely on information from raw text only. nevertheless, all these methods were proposed for a single language 1 and cannot be easily applied to transfer learning in a zero-shot crosslingual setting. ## approach notation let x 1:@xmath0:m ) be contextualised word representations for some input text x 1:m with m words, and computed with a pretrained mmlm. let x n:k (@xmath1, @xmath2:k consisting of k -n words. in our working example we use a single hyperlink x n:k for simplicity, but in practice there may be multiple hyperlinks in the input x 1:m . ## experimental setup we use xlm-r-large @xcite as our mmlm, which is pretrained on a large volume of raw multilingual corpora using mlm training. ## results and discussion results on xl-wsd and xl-wic tasks (tables 1 and 2 ) suggest that our models have a better grasp of word-level semantics than xlm-r, which does not have explicit semantic signals during its pretraining. this is consistent across languages and hyperlink prediction architectures, also when compared to the baseline xlm-r additionally finetuned using mlm training on in-domain wikipedia data. our best models outperform the baselines in both tasks by several points. interestingly, training on in training on the english wikipedia only can be surprisingly effective at times (tables 2 and 3 ), and training on 100 languages shows more consistent improvements only on xl-wic but fails to lead to similar improvements on other tasks. we note that performance on xl-wsd is similar when using 15 or 100 languages, while our evaluation using xtreme shows that performance is slightly worse when using 100 languages compared to using 15 languages only. we conjecture this could be due to the fact we finetune only the last two layers of xlm-r (see appendix b), so the model retains most of the multilingual knowledge it learned during pretraining @xcite . we also hypothesise that the english wikipedia size (in number of words) and quality (in coverage of our hyperlink vocabulary) may also be a reason why training solely on english already brings large gains in transfer to other tasks. for comparison, the english wikipedia is the one with the most data, i.e., about 73m hyperlinks, where the second highest resource language is german with only about 28m hyperlinks (see limitations finally, we highlight that: (1) we report results using single model runs, therefore we have no estimates of the variance of these models; ## conclusions and future work we presented a multilingual wikipedia hyperlink prediction intermediate task to improve the pretraining of contextualised word embedding models. we trained three model variants on different sets of languages, finding that injecting multilingual semantic knowledge consistently improves performance on several zero-shot crosslingual tasks. as future work, we plan to devise a solution to allow crosslingual transferability to scale more efficiently with the number of languages. finally, we will investigate the impact on resource-poor vs resource-rich languages, and the effect of the size and coverage of hyperlinks in model transferability. in proceedings of 11th workshop on building and using comparable corpora, pages 39-42.