Multilingual LAMA : Investigating Knowledge in Multilingual Pretrained Language Models (2021)

Abstract: Recently, it has been found that monolingual English language models can be used as knowledge bases. Instead of structural knowledge base queries, masked sentences such as “Paris is the capital of [MASK]” are used as probes. We translate the established benchmarks TREx and GoogleRE into 53 languages. Working with mBERT, we investigate three questions. (i) Can mBERT be used as a multilingual knowledge base? Most prior work only considers English. Extending research to multiple languages is important for diversity and accessibility. (ii) Is mBERT’s performance as knowledge base language-independent or does it vary from language to language? (iii) A multilingual model is trained on more text, e.g., mBERT is trained on 104 Wikipedias. Can mBERT leverage this for better performance? We find that using mBERT as a knowledge base yields varying performance across languages and pooling predictions across languages improves performance. Conversely, mBERT exhibits a language bias; e.g., when queried in Italian, it tends to predict Italy as the country of origin.

## introduction pretrained language models (lms) @xcite @xcite can be finetuned to a variety of natural language processing (nlp) tasks and generally yield high performance. increasingly, these models and their generative variants are used to solve tasks by simple text generation, without any finetuning @xcite . this motivated research on how much knowledge is contained in lms: @xcite used models pretrained with masked language to answer fill-in-the-blank templates such as "paris is the capital of @xcite ." in summary our contributions are: i) we automatically create a multilingual version of trex and googlere covering 53 languages. ii) we use an alternative to fill-in-the-blank querying -ranking entities of the type required by the template (e.g., cities) -and show that it is a better tool to investigate knowledge captured by pretrained lms. iii) we show that mbert answers queries across languages with varying performance: it works reasonably for 21 and worse for 32 languages. iv) we give evidence that the query language affects results: a query formulated in italian is more likely to produce italian entities (see ## results and discussion we first investigate tyq and untyq and find that tyq is better suited for investigating knowledge in lms. after exploring the translation quality, we use tyq on mlama and observe rather stable performance for 21 and poor performance for 32 languages. when investigating the languages more closely, we find that prediction results highly depend on the language. finally, we validate our initial hypothesis that mbert can leverage its multilinguality by pooling predictions: pooling indeed performs better. ## conclusion we presented mlama, a dataset to investigate knowledge in language models (lms) in a multilingual setting covering 53 languages. while our results suggest that correct entities can be retrieved for many languages, there is a clear performance gap between english and, e.g., japanese and thai. this suggests that mbert is not storing entity knowledge in a language-independent way. experiments investigating language bias confirm this finding. we hope that this paper and the dataset we publish will stimulate research on investigating knowledge in lms multilingually rather than just in english.