m LUKE : T he Power of Entity Representations in Multilingual Pretrained Language Models (2022)

Abstract: Recent studies have shown that multilingual pretrained language models can be effectively improved with cross-lingual alignment information from Wikipedia entities. However, existing methods only exploit entity information in pretraining and do not explicitly use entities in downstream tasks. In this study, we explore the effectiveness of leveraging entity representations for downstream cross-lingual tasks. We train a multilingual language model with 24 languages with entity representations and showthe model consistently outperforms word-based pretrained models in various cross-lingual transfer tasks. We also analyze the model and the key insight is that incorporating entity representations into the input allows us to extract more language-agnostic features. We also evaluate the model with a multilingual cloze prompt task with the mLAMA dataset. We show that entity-based prompt elicits correct factual knowledge more likely than using only word representations.

## introduction pretrained language models have become crucial for achieving state-of-the-art performance in modern natural language processing. in particular, multilingual language models @xcite @xcite have attracted considerable attention particularly due to their utility in cross-lingual transfer. in zero-shot cross-lingual transfer, a pretrained encoder is fine-tuned in a single resource-rich language (typically english), and then evaluated on other languages never seen during fine-tuning. a key to solving cross-lingual transfer tasks is to obtain representations that generalize well across languages. several studies aim to improve multilingual models with cross-lingual supervision such as * work done as an intern at studio ousia. another source of such information is the crosslingual mappings of wikipedia entities (articles). wikipedia entities are aligned across languages via inter-language links and the text contains numerous entity annotations (hyperlinks). with these data, models can learn cross-lingual correspondence such as the words tokyo (english) and 東 京 (japanese) refers to the same entity. wikipedia entity annotations have been shown to provide rich cross-lingual alignment information to improve multilingual language models @xcite . however, previous studies only incorporate entity information through an auxiliary loss function during pretraining, and the models do not explicitly have entity representations used for downstream tasks. in this study, we investigate the effectiveness of entity representations in multilingual language models. entity representations are known to enhance language models in mono-lingual settings @xcite @xcite @xcite presumably by introducing real-world knowledge. we show that using entity representations facilitates cross-lingual transfer by providing languageindependent features. to this end, we present a multilingual extension of luke @xcite . the model is trained with the multilingual masked language modeling (mlm) task as well as the masked entity prediction (mep) task with wikipedia entity embeddings. we investigate two ways of using the entity representations in cross-lingual transfer tasks: (1) perform entity linking for the input text, and append the detected entity tokens to the input sequence. the entity tokens are expected to provide languageindependent features to the model. we evaluate this approach with cross-lingual question answering (qa) datasets: xquad @xcite and mlqa @xcite ; (2) use the entity @xcite token from the mep task as a languageindependent feature extractor. in the mep task, word tokens in a mention span are associated with an entity [mask] token, the contextualized representation of which is used to train the model to predict its original identity. here, we apply similar input formulations to tasks involving mention-span classification, relation extraction (re) and named entity recognition (ner): the attribute of a mention or a pair of mentions is predicted using their contextualized entity [mask] feature. we evaluate this approach with the relx @xcite and conll ner @xcite datasets. the experimental results show that these entitybased approaches consistently outperform wordbased baselines. our analysis reveals that entity representations provide more language-agnostic features to solve the downstream tasks. we also explore solving a multilingual zero-shot cloze prompt task @xcite with the entity [mask] token. recent studies have shown that we can address various downstream tasks by querying a language model for blanks in prompts @xcite . typically, the answer tokens are predicted from the model's word-piece vocabulary but here we incorporate the prediction from the entity vocabulary queried by the entity [mask] token. we evaluate our approach with the mlama dataset @xcite in various languages and show that using the entity [mask] token reduces language bias and elicits correct factual knowledge more likely than using only the word [mask] token. ## adding entities as language-agnostic features in qa we evaluate the approach of adding entity embeddings to the input of mluke-e with cross-lingual extractive qa tasks. the task is, given a question and a context passage, to extract the answer span from the context. the entity embeddings provide language-agnostic features and thus should facilitate cross-lingual transfer learning. ## cloze prompt task with entity representations in this section, we show that using the entity representations is effective in a cloze prompt task @xcite with the mlama dataset @xcite . the task is, given a cloze template such as "[x] was born in [y]" with [x] filled with an entity (e.g., mozart), to predict a correct entity in [y] (e.g., austria). we adopt the typed querying setting @xcite , where a template has a set of candidate answer entities and the prediction becomes the one with the highest score assigned by the language model. model. as in @xcite , the word-based baseline models compute the candidate score as the log-probability from the mlm classifier. when a candidate entity in [y] is tokenized into multiple tokens, the same number of the word [mask] tokens are placed in the input sequence, and the score is computed by taking the average of the logprobabilities for its individual tokens. on the other hand, mluke-e computes the logprobability of the candidate entity in [y] with the entity [mask] token. each candidate entity is associated with an entity in mluke's entity vocabulary via string matching. the input sequence has the entity [mask] token associated with the word [mask] tokens in @xcite , and the candidate score is computed as the log-probability from the mep classifier. we also try additionally appending the entity token of [x] to the input sequence if the entity is found in the vocabulary. to accurately measure the difference between word-based and entity-based prediction, we restrict the candidate entities to the ones found in the entity vocabulary and exclude the questions if their answers are not included in the candidates (results with full candidates and questions in the dataset are in appendix g). results. we experiment in total with 16 languages which are available both in the mlama dataset and the mluke's entity vocabulary. here we only present the top-1 accuracy results from 9 languages on the mluke-w base model lags behind xlm-r base + extra training by 1.7 average points but we can see 5.4 points improvement from xlm-r base + extra training to mluke-e base ([y]), indicating entity representations are more suitable to elicit correct factual knowledge from mluke than word representations. adding the entity corresponding to [x] to the input (mluke-e base ([x] & [y])) further pushes the performance by 11.7 points to 44.9 %, which further demonstrates the effectiveness of entity representations. analysis of language bias. @xcite notes that the prediction of mbert is biased by the input language. for example, when queried in italian (e.g., "[x] e stato creato in @xcite ."), the model tends to predict entities that often appear in italian text (e.g., italy) for any question to answer we qualitatively assess the degree of language bias in the models looking at their incorrect predictions. we show the top incorrect prediction for the template "[x] was founded in @xcite ." for each model in the examples show that the different models exhibit bias towards different entities as in english and french, although in japanese the model consistently tends to predict japan. looking at the degree of language bias, mluke-e base ([x] & [y]) exhibits lower top-1 incorrect ratios overall (27% in fr, 44% in ja, and 30% in fr), which indicates using entity representations reduces language bias. however, lower language bias does not necessarily mean better performance: in french (fr), mluke-e base ([x] & [y]) gives a lower top-1 incorrect ratio than mbert (30% vs. 71%) but their numbers of total false predictions are the same (895). language bias is only one of several factors in the performance bottleneck. ## conclusion we investigated the effectiveness of entity representations in multilingual language models. our pretrained model, mluke, not only exhibits strong empirical results with the word inputs (mluke-w) but also shows even better performance with the entity representations (mluke-e) in cross-lingual transfer tasks. we also show that a cloze-promptstyle fact completion task can effectively be solved with the query and answer space in the entity vocabulary. our results suggest a promising direction to pursue further on how to leverage entity representations in multilingual tasks. also, in the current model, entities are represented as individual vectors, which may incur a large memory footprint in practice. one can investigate an efficient way of having entity representations.