Language Representation Projection: Can We Transfer Factual Knowledge across Languages in Multilingual Language Models? (2023)

Abstract: Multilingual pretrained language models serve as repositories of multilingual factual knowledge. Nevertheless, a substantial performance gap of factual knowledge probing exists between high-resource languages and low-resource languages, suggesting limited implicit factual knowledge transfer across languages in multilingual pretrained language models. This paper investigates the feasibility of explicitly transferring relatively rich factual knowledge from English to non-English languages. To accomplish this, we propose two parameter-free \textbf{L} anguage \textbf{R} epresentation \textbf{P} rojection modules (LRP2). The first module converts non-English representations into English-like equivalents, while the second module reverts English-like representations back into representations of the corresponding non-English language. Experimental results on the mLAMA dataset demonstrate that LRP2 significantly improves factual knowledge retrieval accuracy and facilitates knowledge transferability across diverse non-English languages. We further investigate the working mechanism of LRP2 from the perspectives of representation space and cross-lingual knowledge neuron.

## introduction previous studies demonstrate that a language model is a knowledge base that can recall factual knowledge without additional fine-tuning @xcite . this task of factual knowledge probing, aiming to examine what factual knowledge language models capture during the pre-training phase, can be extended to multiple languages in multilingual pretrained language models, e.g., mbert @xcite , xlm (conneau and lample, 2019), mt5 @xcite , xglm @xcite and bloom @xcite . although multilingual pretrained models serve as repositories of multilingual factual knowledge, a factual knowledge gap exists * corresponding author. many works on cross-lingual transfer @xcite @xcite validate the effectiveness of cross-lingual alignment of representation spaces in facilitating cross-lingual knowledge transfer. these studies primarily evaluate their methods on specific downstream tasks, including natural language inference @xcite , sentence retrieval @xcite , question answering @xcite ) and text generation @xcite , etc. different from such studies, we focus on the task of factual knowledge probing in multilingual pretrained language models and attempt to answer a question in this paper: can cross-lingual alignment of representation spaces enable factual knowledge transfer across languages? in particular, we explore the feasibility of transferring factual knowledge from english to non-english languages. to answer this question, we propose lrp2, which incorporates two parameter-free language representation projection modules into multilingual pretrained models: a language-independent representation projection module that projects representations of non-english languages into englishlike representations and a language-specific representation projection module that maps the englishlike representations back to representations of individual non-english languages. these two modules, as depicted in figure 1 , locate at different layers of transformer. experiments on mlama @xcite ) suggest that lrp2 improves factual knowledge retrieval accuracy and facilitates knowledge transfer across diverse languages. we further conduct indepth analysis to investigate the varying degrees of representation alignment required by different non-english languages, as well as the transferabil-ity of different types of factual knowledge. delving into the working mechanism of lrp2, we identify cross-lingual knowledge neurons in multilingual pretrained language models. our contributions are summarized as follows. ## multilingual factual knowledge probing in the multilingual factual knowledge probing task, multilingual pretrained language models take language-specific fill-in-the-blank queries as input, such as "the capital of england is [mask]" in english, or the corresponding chinese question "英 国的首都是[mask]". as a knowledge base, the probed pretrained language model initially encodes the input query, then retrieves its parameterized memory and ultimately predicts an answer with a probability distribution over the vocabulary. the success of factual knowledge transfer across languages relies on a language-independent representation space for different languages to trigger similar memories within the probed multilingual pretrained model and language-specific representations to allow the model to predict tokens in the corresponding language. ## lrp2 the primary objective of lrp2 is to bridge the gap of factual knowledge probing between english and non-english languages by aligning their representation spaces. @xcite demonstrate that it is possible to induce language-neutral representations for a given language, by subtracting its corresponding language vector. the proposed lrp2 draws inspiration from this work and initiates its process by computing a set of language vectors v l for each language l. specifically, for language l, we feed a set of its sentences into the multilingual multi-head attention feed forward lirp lsrp × i multi-head attention feed forward multi-head attention feed forward ×(j-i) ×(l-j) embedding embedding memory access (rich) baseline lrp2 england capital [mask] of is 的 是 英国 首都 [mask] 的 是 英国 首都 [mask] 𝑥 𝑦 𝑥 𝑦 𝑥 𝑦 𝑥 𝑦 英国 首都 的 是 [mask] zh , v i en , v j zh and v j en represent language vectors obtained for chinese and english from the i-th and j-th layer of the multilingual pretrained language model in advance. we use chinese to showcase our framework and our method is applicable to other languages in the same way. for simplicity, we ignore other sublayers in transformer in the diagram. note that our method is based on the core assumption that the representation spaces of two languages can be transferred through a euclidean distance mapping. this form of straightforward mapping is relatively coarse, incapable of achieving the level of precise semantic transfer depicted in the figure, which is presented for the sake of illustration but may appear somewhat overly idealized. as illustrated in figure 1 , lrp2 incorporates two language representation projection modules into the probed multilingual pretrained language model, which are referred to as the language-independent representation projection (lirp) module and the language-specific representation projection (lsrp) module, respectively. these two modules as mentioned in section 2, in the multilingual factual knowledge probing task, it is essential for the multilingual pretrained language model to yield answers in the corresponding language. to recover the language-specific information of the input language, we insert the lsrp into the j-th layer of the model. the back-projection to the input language is formulated as: ## experiments we conducted extensive experiments to examine the effectiveness of the proposed lrp2 framework in factual knowledge transfer across languages. ## working mechanism of lrp2 in this section, we study the working mechanism of lrp2 from the perspectives of representation space and knowledge neuron. ## conclusion we have presented a simple yet effective method to transfer factual knowledge from english to non-english languages in multilingual pretrained language models. we empirically confirm that crosslingual alignment of representation spaces enables factual knowledge transfer across languages in multilingual pretrained language models. further analysis on knowledge neurons shows that the align-ment of english and non-english representation spaces brought by lrp2 can help non-english languages to stimulate knowledge neurons similar to english, thereby acquiring knowledge transferred from english.