Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledge (2024)

Abstract: Acquiring factual knowledge for language models (LMs) in low-resource languages poses a serious challenge, thus resorting to cross-lingual transfer in multilingual LMs (ML-LMs). In this study, we ask how ML-LMs acquire and represent factual knowledge. Using the multilingual factual knowledge probing dataset, mLAMA, we first conducted a neuron investigation of ML-LMs (specifically, multilingual BERT). We then traced the roots of facts back to the knowledge source (Wikipedia) to identify the ways in which ML-LMs acquire specific facts. We finally identified three patterns of acquiring and representing facts in ML-LMs: language-independent, cross-lingual shared and transferred, and devised methods for differentiating them. Our findings highlight the challenge of maintaining consistent factual knowledge across languages, underscoring the need for better fact representation learning in ML-LMs.

## introduction to mitigate the inherent data sparseness of lowresource languages, multi-lingual language models (ml-lms) such as mbert @xcite , xlm-r @xcite , mt5 @xcite , and bloom @xcite have been developed to transfer knowledge across languages. the effectiveness of this cross-lingual transfer in ml-lms has been demonstrated on various language tasks @xcite @xcite . however, a more challenging task is the cross-lingual transfer of specific factual knowledge, such as "greggs is a british bakery chain." in many low-resource languages, text data about such knowledge might be minimal or non-existent. effectively transferring knowledge is vital for applications that handle factual knowledge, such as fact checking and relation extraction @xcite . following early studies @xcite ) that used cloze-style queries to probe whether monolingual language models can recall factual knowledge, researchers probed ml-lms @xcite @xcite @xcite . the results indicated that ml-lms exhibit an ability to recall facts. however, the mechanism behind the acquisition and representation of facts in ml-lms remains unclear. in this study, we investigated whether and how low-resource languages can benefit from the crosslingual transfer of factual knowledge (figure 1 ). concretely, we addressed three research questions: rq1: how does factual probing performance of ml-lms differ across languages, and what factors affect these differences? ( §4) rq2: do ml-lms represent the same fact in different languages with a shared or independent representation? ( §5) rq3: what mechanisms during the pre-training of ml-lms affect the formation of cross-lingual fact representations? ( §6) to answer these research questions, we started by probing two ml-lms, mbert and xlm-r, using the mlama probing dataset @xcite . the results reconfirm that ml-lms have difficulty recognizing facts in low-resource languages @xcite , such as irish and lithuanian ( §3). however, we also observed only a moderate correlation between probing performance and the amount of training data. although the cultural bias of the mlama dataset may hinder probing performance in non-latin script languages (keleg and magdy, 2023), the exact effect of a model's cross-lingual capabilities remains to be established. to identify the role of cross-lingual capability in fact probing, we performed a neuron-level analysis for facts predicted correctly. by comparing active neurons across languages, we observed that identical facts in various languages are not acquired in identical ways. for specific facts, some languages exhibit similar neuron activity, while others display distinct patterns. we categorize the former as crosslingual fact representations, as illustrated in figure 1(b , c ), and the latter as language-independent representations, as illustrated in figure 1(a) . to further discern cross-lingual representations, we devised a method for tracing the roots of facts by verifying their presence in the knowledge source (wikipedia for mbert). we assume that the facts that are predicted correctly although absent in the training data are captured through cross-lingual transfer, referred to as cross-lingual transferred (figure 1(c )) to differentiate from cross-lingual shared (figure 1(b) ). a deeper investigation into the results, however, revealed that only a fraction of those facts would be acquired through cross-lingual transfer. this underscores the limitations of current ml-lms in cross-lingual fact representation. ## related work this section reviews existing studies on understanding the mechanism of cross-lingual transfer and factual knowledge probing. we first discuss key studies that investigated how knowledge is transferred across languages in ml-lms. next, we highlight research on how factual knowledge is perceived in pre-trained language models (plms). ## what factors influence discrepancy in factual probing across languages? the vertical bars in figure 3 showing the results of factual probing in various languagesreveal substantial differences among languages. in this section, we will evaluate the potential factors for these differences and examine how they relate to proficiency in the cross-lingual transfer ability of ml-lms. ## do ml-lms have fact representations shared across languages? in this section, we discuss how ml-lms represent facts within their parameter spaces by exploring two scenarios. in one scenario, a copy of the same fact is independently maintained in different languages, as illustrated in figure 1(a) ; ml-lms based on this scenario are referred to as "languageindependent." in the other scenario, fact representations in different languages are unified in an embedding space, as illustrated in figure 1(b , c ); ml-lms based on this scenario are referred to as "cross-lingual." the language-independent scenario will hinder cross-lingual transfer in fine-tuning ml-lms on downstream tasks, where the training data is available in a few languages. ## conclusions our research establishes the groundwork for further studies in understanding cross-lingual factual knowledge representation. through comprehensive factual probing experiments and analysis for 53 languages using the mbert multilingual language model, we evaluated key factors in the differences between their proficiencies in cross-lingual transfer of factual knowledge, such as the training data volume and mask token count, and identified knowledge sharing patterns among geographically proximate language clusters. we leverage the existing neuron probing and the proposed knowledge tracing methods to identify three types of patterns for acquiring and representing factual knowledge across languages in ml-lms: language-independent, cross-lingual shared, and cross-lingual transferred. analysis revealed the challenges involved in achieving effective crosslingual transfer of factual knowledge from highresource to low-resource languages in ml-lms. future work aims to enhance the cross-lingual fact representation learning in ml-lms and develop a more precise factual probing dataset.