Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Review (2023)

Abstract: In recent years, pre-trained Multilingual Language Models (MLLMs) have shown a strong ability to transfer knowledge across different languages. However, given that the aspiration for such an ability has not been explicitly incorporated in the design of the majority of MLLMs, it is challenging to obtain a unique and straightforward explanation for its emergence. In this review paper, we survey literature that investigates different factors contributing to the capacity of MLLMs to perform zero-shot cross-lingual transfer and subsequently outline and discuss these factors in detail. To enhance the structure of this review and to facilitate consolidation with future studies, we identify five categories of such factors. In addition to providing a summary of empirical evidence from past studies, we identify consensuses among studies with consistent findings and resolve conflicts among contradictory ones. Our work contextualizes and unifies existing research streams which aim at explaining the cross-lingual potential of MLLMs. This review provides, first, an aligned reference point for future research and, second, guidance for a better-informed and more efficient way of leveraging the cross-lingual capacity of MLLMs.

## introduction the objective of cross-lingual transfer is to leverage knowledge learned by a model in a source language and to transfer it to a target language. while such a process of transferring knowledge and concepts across languages seems natural for a polyglot, it is believed to be less straightforward for a language model. nevertheless, multilingual language models (mllms), such as mbert @xcite , xlm @xcite and xlm-r @xcite demonstrate effective crosslingual transfer capabilities. such a transfer ability is moderately expected from xlm, given that par- * research was conducted at zortify. allel data is leveraged through a cross-lingual transfer learning objective during pre-training. however, it is less anticipated for mbert and xlm-r, which are pre-trained on separate monolingual corpora without any explicit cross-lingual signal. nevertheless, the latter show a surprisingly strong cross-lingual transfer capacity on a variety of downstream tasks @xcite . while no apparent factors explaining the nature of this ability can be intuitively derived from the properties of mllms, there have been many attempts to understand this behavior. past research has outlined and investigated various factors that may impact cross-lingual transfer performance in mllms, but there are still open questions due to conflicting findings across studies. in our work, we inspect findings from past research investigating the inner workings of crosslingual transfer in mllms. we not only outline overlapping contributions with consensual findings but also highlight and attempt to resolve conflicts between contradictory studies. our work is structured according to five different types of factors whose impact on cross-lingual transfer capacity has been investigated in the past: the examination of these factors provides insight into how and why mllms perform differently in different contexts. this understanding contributes to the overall explainability of mllms, which is essential for efficiently leveraging their cross-lingual transfer capacities and improving their performance in general. a list of all the papers surveyed in this study is provided in appendix a. ## related work recently, numerous studies have investigated how to leverage the cross-lingual potential of mllms for better transfer among languages. @xcite conducted a survey on existing cross-lingual transfer paradigms but did not investigate the components that are responsible for their inner workings. @xcite , in their survey on pre-trained mllms, commented on various factors that affect cross-lingual transfer. since they discussed a wide range of topics, they could not investigate in depth the findings from the studies that examined these factors. after the publication of that work, many studies have further investigated various factors that impact transfer performance and have helped to resolve some of the conflicts among past contributions. @xcite introduced a linguistic blood bank that shows that not all languages transfer equally well among each other. this emphasizes the need for a clearer understanding of the underlying factors that contribute to this imbalance. on a related note, @xcite found that english is not the overall best source language for cross-lingual transfer, despite its dominance in the pre-training corpus. hence, automating the process of selecting a source language for cross-lingual transfer has been pursued on many occasions @xcite @xcite . these attempts focused on creating meta-models foot_10 which aim to predict the most suitable source language for a given use-case based on some of the factors from section 3. by incorporating typological features, ansell et al. (2021), lee et al. (2022) and chronopoulou et al. (2023) enhanced the performance of adapters for low-resource languages. however, our survey reveals that adapters and other methods could benefit from more than just typological factors when dealing with low-resource scenarios. ## discussion building on previous research, our study investigated various factors that impact cross-lingual transfer performance. we examined a range of factors, including language-related factors as well as factors related to the models and training data. one of the existing challenges is the presence of contradictory findings from past studies. to better understand these discrepancies, we outlined possible explanations that could account for these differences, including the varying implementation details of experiments and evaluation methods. one of the key variations among the various studies is the use of synthetic and natural languages. synthetic languages can be created with a controlled level of variation by manipulating specific linguistic features. however, they may not capture the full range of complexity found in natural languages, which may limit their usefulness in drawing conclusions that apply to real-world settings. while we acknowledge the value of the efficiency of using transfer performance prediction models to automate the selection of transfer languages, the accuracy of relying on feature importance values to make conclusions about the individual impact of specific factors on cross-lingual transfer performance cannot be taken as an absolute. our survey results show that all the factors we examined affect cross-lingual transfer in different ways and settings. although the interaction of factors has only been investigated in a limited number of past studies, our findings suggest that some factors can influence the importance of others. additionally, there is evidence suggesting that there are task-specific differences, for example, the pre-training corpus size being more important for higher-level tasks and lexical overlap, and word order being more important for lower-level tasks. therefore, we strongly encourage future research to examine the full range of interactions among different factors as well as the underlying reasons for task-specific divergences. given that especially linguistic features have been shown to have a strong impact on crosslingual transfer performance, we suggest that future research could examine whether languages are indeed the most suitable basis for constructing multilingual models. instead of focusing on the distribution of languages in the pre-training corpus, it might be more efficient to focus on the distribution of linguistic features. one possible approach is to cluster texts according to their syntactic complexity or their morphological diversity, irrespective of their language affiliation. this would enable the development of a model that could potentially better transfer to languages that were absent in the pretraining corpus but which share linguistic features with the languages that the model has seen during pre-training. in addition, we advocate for the development of more multilingual downstream task datasets that encompass a wider and more diverse range of languages, as this would enable a more comprehensive and robust assessment of cross-lingual transfer capabilities across various language models and approaches. furthermore, we urge more investigation on the influence of the aforementioned factors on generative models, as this area remains relatively unexplored despite the current prominence of gpt-like models.