Analysis of Multi-Source Language Training in Cross-Lingual Transfer (2024)

Abstract: The successful adaptation of multilingual language models (LMs) to a specific language-task pair critically depends on the availability of data tailored for that condition. While cross-lingual transfer (XLT) methods have contributed to addressing this data scarcity problem, there still exists ongoing debate about the mechanisms behind their effectiveness.In this work, we focus on one of promising assumptions about inner workings of XLT, that it encourages multilingual LMs to place greater emphasis on language-agnostic or task-specific features. We test this hypothesis by examining how the patterns of XLT change with a varying number of source languages involved in the process.Our experimental findings show that the use of multiple source languages in XLT-a technique we term Multi-Source Language Training (MSLT)-leads to increased mingling of embedding spaces for different languages, supporting the claim that XLT benefits from making use of language-independent information. On the other hand, we discover that using an arbitrary combination of source languages does not always guarantee better performance. We suggest simple heuristics for identifying effective language combinations for MSLT and empirically prove its effectiveness.

## introduction there has been recent interest in multilingual language models (lms), where multiple languages are used in training an lm, due to its capability of accepting multilingual inputs and achieving similar or even better performance compared to monolingual lms at natural language processing (nlp) tasks. however, despite all these advantages, fine-tuning multilingual lms using a specific language and an nlp task hinges on acquiring labeled data tailored to a specific purpose, which is not always possible. to address this challenge amid linguistic resource disparities, cross-lingual transfer (xlt) approaches have been proposed. in the context of finetuning multilingual lms, xlt leverages resourcerich source languages in fine-tuning to boost performance on low-resource target languages. although there is still active debate regarding the inner workings of xlt, a significant body of previous research indicates that multilingual lms are capable of separating language-specific and language-agnostic information from text @xcite , and that xlt favors the enhancement of language-agnostic features while reducing the emphasis on language-specific ones @xcite @xcite . while the standard practice in xlt is to employ only one language as a source for transfer, there exist attempts that have investigated the use of multiple source languages @xcite @xcite @xcite . these studies reveal that employing multiple source languages in xlt-the concept we refer to as multi-source language ## 712 training (mslt)-leads to performance improvement. however, despite it is evident that mslt has clear advantages, several research questions remain unexplored, including: (1) the specific changes that it induces in the internal states of lms, (2) the conditions under which it achieves its optimal performance, and (3) the breadth of applications where it can be effectively applied, among others. in this work, we introduce a series of comprehensive analyses exploring the inner workings of mslt, along with offering guidance for its effective utilization. specifically, we first attempt to illustrate the utility of mslt with a couple of intuitive visualizations, to clarify the underlying reasons for its effectiveness. in addition, our experimental results discover various intriguing findings. regarding the number of languages involved in mslt, we observe its positive correlation with performance in general, although the trend tends to plateau as the number of languages exceeds a certain threshold. our empirical discovery also suggests that the enhanced diversity of source languages in mslt does not invariably lead to improved performance. this underscores the importance of carefully selecting an effective combination of source languages (see figure 1 for an intuitive illustration). therefore, we examine various heuristics that can identify a suitable selection of source languages from an exponentially large pool of possible combinations. based on our tests, we propose several competent strategies, some of which are derived from the statistical properties of languages in terms of pretraining, while others utilize linguistic characteristics of languages. lastly, we discuss intriguing patterns evident in the group of languages selected as sources for transfer, providing insights into the interplay of languages in mslt. ## related work multilingual lms @xcite ; inter alia) are widely recognized for their ability to process inputs from diverse languages in an integrated manner. they have recently been the subject of extensive research for uncovering their working mechanisms, and this effort has converged on the robust hypothesis that the internal states of multilingual lms can be categorized into language-sensitive and language-agnostic components @xcite @xcite . meanwhile, cross-lingual transfer (xlt) aims to improve the performance of multilingual lms on specific tasks in languages with limited resources (i.e., target languages), by utilizing the support from resource-rich languages (i.e., source languages). previous research on xlt has explored diverse implementations of the concept. for instance, @xcite attempt to achieve xlt by blending representations from source and target languages, while @xcite introduce data augmentation techniques such as subword sampling and code-switching. @xcite propose a contrastive learning framework aimed at reducing the differences in sentence embeddings between languages. concurrently, @xcite present a prompt-based approach that facilitates information transfer from a source language to prompts composed in the target language. note that the majority of existing methods for xlt @xcite all available languages without specifically seeking to determine the ideal number of languages for the optimal performance of mslt. our work tackles these limitations, by presenting a series of wellsupported illustrations that clarify how and why mslt operates, along with providing guidelines for enhanced utilization of the mslt framework. ## terminology we first clarify the core concepts and their corresponding acronyms in this section. cross-lingual transfer (xlt) xlt is a technique applicable to multilingual lms, particularly when obtaining a satisfactory amount of data for the language-task pair of interest is challenging. this is built on the assumption that fine-tuning a multilingual model for a task t in a resource-rich language a can lead to a synergistic improvement in performance for another language b, especially if b has limited data available for the task t . in this work, we focus on the zero-shot xlt setting, under the assumption that no suitable data is available for the target language. single-source language training (sslt) sslt represents an environment for xlt where multilingual lms are trained on a single language. from now on, we adopt the notation 'sslt(l)' to signify the setting where the language l is exclusively utilized as a source language in xlt. ## verifying the effectiveness of mslt we present a thorough analysis to confirm the effectiveness of mslt, and additionally provide insights into its inner workings. we describe the intuition behind how mslt enhances xlt performance ( §4.2) and present visualizations related to our intuition ( §4.3). we also provide empirical findings with respect to the ideal number of source languages in mslt ( §4.4). section 4.1 presents the configurations used throughout the section. ## language set composition in mslt in the previous section, we demonstrated that mslt exhibits markedly superior performance on xlt compared to sslt. assuming the selection of k source languages from a total of n languages, there exist ( n k ) possible combinations. given that n can be significantly larger than k, fine-tuning with every possible combination of source languages to identify the optimal mix is not time-and resourceefficient. therefore, it is necessary to select an appropriate combination more efficiently. in this section, we first establish the importance of selecting an effective set of source languages and then proceed to examine various criteria for choosing an optimal source language combination. ## conclusion this work demonstrates the positive impact of multi-source language training (mslt) in crosslingual transfer (xlt). by leveraging multiple source languages, mslt facilitates the learning of language-agnostic features in multilingual lms. we provide qualitative and quantitative evidence suggesting that this approach promotes crosslingual semantic alignment, leading to better xlt performance compared to using a single source language. furthermore, we identify the importance of selecting the optimal size and composition of source language sets for mslt. to guide this selection efficiently, we propose diverse criteria based on the statistical and linguistic properties of candidate languages, such as lang2vec. importantly, mslt demonstrates effectiveness across various configurations, including different tasks, architectures, and training paradigms. we hope this study serves as a springboard for further exploration of factors influencing the development of effective cross-lingual transfer methodologies.