Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structure (2022)

Abstract: Multilingual pre-trained language models, such as mBERT and XLM-R, have shown impressive cross-lingual ability. Surprisingly, both of them use multilingual masked language model (MLM) without any cross-lingual supervision or aligned data. Despite the encouraging results, we still lack a clear understanding of why cross-lingual ability could emerge from multilingual MLM. In our work, we argue that cross-language ability comes from the commonality between languages. Specifically, we study three language properties: constituent order, composition and word co-occurrence. First, we create an artificial language by modifying property in source language. Then we study the contribution of modified property through the change of cross-language transfer results on target language. We conduct experiments on six languages and two cross-lingual NLP tasks (textual entailment, sentence retrieval). Our main conclusion is that the contribution of constituent order and word co-occurrence is limited, while the composition is more crucial to the success of cross-linguistic transfer.

## introduction zero-shot cross-lingual transfer aims to build models for the target language by reusing knowledge learned from the source language. in this way, models can be efficiently implemented in multilingual as well as low-resource language scenarios. traditionally, it is solved by a two-step pipeline @xcite : a shared multilingual textual representation is first built and then supervised data from the source language is used on the top of it to train task-specific models. with the recent emergence of multilingual language models, the standard paradigm in this field has shifted to the pre-trained fine-tuning paradigm. multilingual pretrained language models, such as mbert (devlin 了 figure 1 : example of two sentences in different languages, which are different in constituent order, but are very similar in constituent tree. note that we simplify the constituent tree for better understanding. @xcite and xlm-r @xcite , have proven effective for cross-lingual transfer with better results on a large number of downstream tasks and languages @xcite . the most surprising part is that mbert and xlm-r are both trained without using any parallel corpus. previous work @xcite attribute this success to the shared anchor words. but recent work @xcite shows that cross-lingual transfer still could emerge even corpus of languages are from different domain, or don't share any common words. for cross-lingual model, sharing transformer encoder weight is critical, while whether having language specific word embedding or language identity marker is not important. this makes us more curious about what kind of common prop-erty of languages could make cross-lingual transfer successful. in our work, we study three language structure properties: 1) constituent order. specifically, we study three common constituent orders: order of verb and object, adposition and noun phrase, adjective and noun. 2) composition. composition means that we could combine two or several simple meanings and build a new more complicated meaning. for example, two words could form a phrase, and more composition could form a sentence recursively. 3) word co-occurrence. we take the bag of words assumption and study the word co-occurrence in a sentence. we use figure 1 to better show the composition similarity between two sentences. although the english sentence and chinese sentence have different word order, but they are both first divided into a noun phrase and a verb phrase, and the verb phrase is then divided into three parts that are identical in meaning but inconsistent in order. to better analyze the contribution of these three properties, we use the control variable method. based on a successful transfer between the source and target languages, we change or remove only one structure property in the source language. we measure the importance of this property by testing the change in performance of the cross-language transfer from the modified source language to the target language. the results show that the effect of constituent order and word co-occurrence is small, while composition has a greater effect. the main contributions are summarized as follows: (1) we analyze the source of cross-linguistic ability from shared properties in language structure and propose three candidate answers. (2) we used the control variable method, modifying only the target property in the corpus and keeping the other settings identical, thus better quantifying the contribution of the studied properties. (3) our experiments clearly show that constituent order and word co-occurrence make very limited contributions to cross-lingual ability, while composition is the key to cross-lingual transfer. ## study design in this section, we introduce the design of our study, including the three language structure properties, and the overall setup. we also detail the pre-training and fine-tuning settings for better reproduction. ## constituent order previous work @xcite has argued that the cross-lingual transfer performance between languages with same constituent order is 10%-20% better than languages with different constituent order. so we further conduct control variate experiments to study the influence of constituent order. first we introduce the constituent order we studied and experiment setup. then we analyze the effects of constituent order through the results. our main conclusion is that the contribution of constituent order is about 1%. ## composition: the key to zero-shot cross-lingual transfer in this section, we study the contribution of constituent order, composition and word co-occurrence respectively. we first present how to completely remove constituent order and composition step-bystep from the corpus, and then analyze the results. subsequently, by controlling the rate of composition retention, we further quantified its contribution to cross-lingual transfer. ## conclusion in this paper, we study the source of cross-lingual ability in the multilingual masked language model in the view of language structure. we study three language structure properties: constituent order, composition and word co-occurrence. the experiments are conducted using control variable method. we create an artificial language by modifying property in source language. we quantify the contribution of these three properties separately through cross-language transfer performance changes from the modified language to the target language. the results show that the contribution of constituent order and word co-occurrence are very limited, while composition is actually the key to cross-lingual transfer. how to use this finding to enhance pretrained multilingual language models and improve performance on cross-lingual nlp tasks will be our focus for future work.