X - FACTR : Multilingual Factual Knowledge Retrieval from Pretrained Language Models (2020)

Abstract: Language models (LMs) have proven surprisingly successful at capturing factual knowledge by completing cloze-style fill-in-the-blank questions such as “Punta Cana is located in _.” However, while knowledge is both written and queried in many languages, studies on LMs’ factual representation ability have almost invariably been performed on English. To assess factual knowledge retrieval in LMs in different languages, we create a multilingual benchmark of cloze-style probes for typologically diverse languages. To properly handle language variations, we expand probing methods from single- to multi-word entities, and develop several decoding algorithms to generate multi-token predictions. Extensive experimental results provide insights about how well (or poorly) current state-of-the-art LMs perform at this task in languages with more or fewer available resources. We further propose a code-switching-based method to improve the ability of multilingual LMs to access knowledge, and verify its effectiveness on several benchmark languages. Benchmark data and code have be released at https://x-factr.github.io .

## introduction language models (lms; @xcite @xcite ) learn to model the probability distribution of text, and in doing so capture information about various aspects of the syntax or semantics of the language at hand. recent works have presented intriguing results demonstrating that modern large-scale lms also capture a significant amount of factual knowledge @xcite @xcite . this knowledge is generally probed by having the lm fill in the blanks of cloze-style prompts such as figure 1 : x-factr contains 23 languages, for which the data availability varies dramatically. prompts get instantiated to produce grammatical sentences with different numbers of mask tokens and are used to obtain predictions for [y]. in this spanish example, the verb gerund "fundar" to found is rendered as "fundada" to agree in gender and number with the subject "bloomberg l.p.". the final prediction is in bold. we create a new multilingual benchmark for probing factual knowledge in lms -the crosslingual factual retrieval benchmark (x-factr). x-factr shares a similar formulation as the lama benchmark of @xcite , which assesses whether lms have memorized a fact (i.e., a subject-relation-object triple) by having lms predict the blank (i.e. object) in a cloze-style prompt for each relation after filling in the subject. we manually create such prompts for 23 languages spanning different language families and different levels of data availability ( § 3.1). because many languages that we handle are morphologically rich, we design a morphology-sensitive annotation schema (see example in fig. 1 ) that can properly instantiate prompts using entity metadata (e.g. gender) and a morphological inflection model ( § 3.3). in addition, while previous works @xcite @xcite have limited examination to single-token entities (e.g. "france"), we expand our setting to include multi-token entities (e.g. "united states"), which comprise more than 75% of facts included in our underlying database (wikidata; § 3.2). we propose several decoding algorithms for prediction of these multi-token entities using masked lms ( § 4). we discuss the related work in depth in § 7. we perform experiments on x-factr ( § 5), comparing and contrasting across languages and lms to answer the following research questions: (1) how and why does performance vary across different languages and models? (2) can multilingual pre-training increase the amount of factual knowledge in lms over monolingual pre-training? (3) how much does knowledge captured in different languages overlap? we find that the factual knowledge retrieval of m-lms in high-resource languages is easier than in low-resource languages, but the overall performance is relatively low, indicating that this is a challenging task. we analyze the types of failure cases, shedding light on future directions to improve factual knowledge in m-lms. in addition, multilingual pre-training does not necessarily lead to a higher recall of facts compared to language-specific monolingual pre-training. the knowledge memorized by m-lms in fact is largely distinct across languages, with almost 50% of facts being recalled in only one language. inspired by the above observations, we propose a code-switching-based objective function to improve the ability of m-lms to access knowledge using queries from a variety of languages. we replace entities in a sentence from the original language with counterparts in another lan-guage, and further fine-tune the lm on these codeswitched data ( § 6). we perform experiments on three languages (french, russian, and greek, codeswitched with english). results demonstrate that this code-switching-based learning can successfully improve the knowledge retrieval ability with low-resource language prompts. ## retrieving facts from lms in this paper we follow the protocol of @xcite 's english-language lama benchmark, which targets factual knowledge expressed in the form of subject-relation-object triples from wikidata foot_0 curated in the t-rex dataset @xcite . the cloze-style prompts used therein are manually created and consist of a sequence of tokens, where [x] and [y] are placeholders for subjects and objects (e.g. "[x] is a [y] by profession."). to assess the existence of a certain fact, [x] is replaced with the actual subject (e.g. "obama is a mask by profession.") and the model predicts the object in the blank ŷi = argmax y i p(y i |s i:i ), where s i:i is the sentence with the i-th token masked out. finally, the predicted fact is compared to the ground truth. in the next section, we extend this setting to more languages and predict multiple tokens instead of a single one. ## multi-token decoding as tab. 1 shows, many facts involve multi-token entities and thus a lm would need to predict these entities in multiple steps. generating multiple predictions is straightforward for traditional left-toright lms @xcite , where we can autoregressively decode the next token conditioned on previous tokens. however, many pre-trained lms such as bert @xcite are masked lms that predict individual words given left and right contexts, and decoding from such masked lms remains an open problem @xcite @xcite @xcite . we systematically examined different multi-token decoding algorithms from three orthogonal perspectives: (1) how the initial predictions are produced, (2) how to refine the predictions, and (3) other commonly used components in neural text generation systems. we assume that the following conditional probability distribution is defined by the masked lm for a sentence with n tokens: ## x-factr benchmark performance implementation details. we use the implementations of different multilingual/monolingual pretrained lms in the transformers library @xcite . we examine 3 multilingual pre-trained lms, m-bert, xlm, xlm-r @xcite @xcite , 5 and 8 monolingual pre-trained lms, bert (en), camembert (fr), bertje (nl), beto (es), ru-bert (ru), chinese bert (zh), berturk (tr), and greekbert (el) @xcite @xcite @xcite . details of these models can be found in appendix d. we set the maximal number of mask tokens to @xmath0, french, dutch, and spanish. in these languages more than 90% of the entities are split @xmath1. for all other languages we use @xmath2. this is expected because the vocabulary of m-lms based on wordpiece tokenization is dominated by frequent words and low-resourcelanguage words tend to split into more pieces @xcite . we set the maximal number of iterations to @xmath3, so that we can approximately refine all the predicted tokens once for a sentence with 5 yoruba is not in the training data of xlm and xlm-r. m mask tokens (the initial prediction takes exactly m iterations). in our main results, we report results with two decoding algorithms: the simplest independent generation method and the confidencebased method for both initial and refinement predictions. the latter performs better than order-based methods, as we will show in tab. 3. to save computation time, we only use confidence re-computation for @xmath4. we discuss computation complexity in appendix c. evaluation metrics. we follow @xcite , computing the accuracy of predicted objects for each relation and macro-average them as final scores. for fine-grained analysis of different decoding methods, pre-trained lms, and languages, we report results on all facts as well as on subsets consisting only of single-token objects (single) and multi-token objects (denoted as multi). ## improving multilingual lm retrieval as the performance of m-lms is relatively low, especially on low-resource languages, an obvious endeavor is to refine the model to improve fact retrieval performance in various languages. we analyze how similarly m-bert performs on queries in different languages. we collect correctly predicted facts across all languages, and count in how many languages each fact was retrieved correctly. as shown in the bottom-left histogram of fig. 5 , half of the correctly predicted facts were correct in a single language, indicating little overlap across languages @xcite . only 3% of facts were correct in more than 5 languages, and objects in those facts are usually sub-strings of subjects, making them easy to retrieve regardless of the language. this observation is also confirmed by the overlap between pairs of languages in the top-right chart of fig. 5 ; even the most similar languages (i.e., english and dutch) only have 34% of correct predictions in common. we find that facts retrievable only in a single language tend to be knowledge that is mainly mentioned in a certain language. for example, m-bert mistakenly predicts "qq" in the english sentence "tencent qq is developed by _.", while the prediction "腾讯" (tencent) in the corresponding chinese sentence "腾讯qq是由_开发的。" is correct. this is probably because tencent, a chinese company, is more frequently mentioned in the chinese training corpus. ## related work factual knowledge retrieval from lms several works have focused on probing factual knowledge solely from pre-trained lms without access to external knowledge. they do so by either using prompts and letting the lm fill in the blanks, which assumes that the lm is a static knowledge source @xcite @xcite , or fine-tuning the lm on a set of question-answer pairs to directly generate answers, which dynamically adapts the lm to this particular task @xcite . impressive results demonstrated by these works indicate that large-scale lms contain a significant amount of knowledge, in some cases even outperforming competitive question answering systems relying on external resources @xcite . @xcite further shows that lms can generate even more factual knowledge when augmented with retrieved sentences. our work builds on these works by expanding to multilingual and multi-token evaluation, and also demonstrates the significant challenges posed by this setting. multilingual benchmarks many multilingual benchmarks have been created to evaluate the performance of multilingual systems on different natural language processing tasks, including question answering @xcite @xcite , natural language understanding @xcite @xcite , syntactic prediction @xcite , and comprehensive benchmarks covering multiple tasks @xcite . we focus on multilingual factual knowledge retrieval from lms, which to our knowledge has not been covered by any previous work. ## conclusion we examine the intersection of multilinguality and the factual knowledge included in lms by creating a multilingual and multi-token benchmark x-factr, and performing experiments comparing and contrasting across languages and lms. the results demonstrate the difficulty of this task, and that knowledge contained in lms varies across languages. future directions include other pre-training or fine-tuning methods to improve retrieval performance and methods that encourage the lm to predict entities of the right types.