Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Models (2024)

Abstract: Recent advances in training multilingual language models on large datasets seem to have shown promising results in knowledge transfer across languages and achieve high performance on downstream tasks. However, we question to what extent the current evaluation benchmarks and setups accurately measure zero-shot cross-lingual knowledge transfer. In this work, we challenge the assumption that high zero-shot performance on target tasks reflects high cross-lingual ability by introducing more challenging setups involving instances with multiple languages. Through extensive experiments and analysis, we show that the observed high performance of multilingual models can be largely attributed to factors not requiring the transfer of actual linguistic knowledge, such as task- and surface-level knowledge. More specifically, we observe what has been transferred across languages is mostly data artifacts and biases, especially for low-resource languages. Our findings highlight the overlooked drawbacks of existing cross-lingual test data and evaluation setups, calling for a more nuanced understanding of the cross-lingual capabilities of multilingual models.

## introduction massively multilingual transformers (mmts) exhibit remarkable abilities in comprehending texts in multiple languages @xcite @xcite . their performance on extensive multilingual benchmarks @xcite including natural language inference @xcite , paraphrase identification @xcite , question answering @xcite , and commonsense reasoning @xcite , clearly indicates their usefulness for downstream tasks across different languages. since collecting labeled data for multiple languages is expensive, fine-tuning an mmt on only one language (usually english) and applying it to other languages is a common practice to expand multilingual models' applications to more languages. the success of employing this approach is primarily attributed to the cross-lingual ability of mmts and knowledge transfer across languages @xcite @xcite . however, in this work, we question this attribution and take a closer look at the cross-lingual evaluation pipelines used to assess mmts' performance. our study aims to broaden our horizons about the existing performance-based evaluation setups of mmts and highlight their significant shortcomings. to this end, we first provide insights into the definition of the cross-lingual ability in language models and the essential criteria for its evaluation. we challenge the assumption of high cross-lingual ability, i.e., requiring actual linguistic knowledge, in mmts using three downstream tasks: natural language inference (nli), paraphrase identification (pi), and question answering (qa). our experimental results demonstrate that multilingual models struggle with transferring linguistic knowledge across languages when the inputs involve multiple languages, such as in the nli task with a premise in arabic and a hypothesis in spanish. employing this setup, we show that, unlike previous assumptions, mmts are not able to effectively connect the underlying semantics between languages in a zero-shot manner. it is worth mentioning that the evaluation of mmts using multiple languages in the input offers both theoretical advantages and reflects real-world scenarios in nlp systems. we extend our study to investigate if the failures come from the lack of across-language fine-tuning data. we find that even by fine-tuning mmts on across-language data that involves two languages in an instance, they still can not successfully trans-fer knowledge between languages in a zero-shot setting. looking for the reasons behind the ineffectiveness of mmts on the across setup, we examine the impact of individual samples on the cross-lingual performance and identify a specific subset where mmts struggle. our findings demonstrate that models achieve exaggerated high performance by strongly relying on spurious features and data artifacts. we notice that cross-lingual transfer primarily involves learned biases and shallow knowledge rather than linguistic knowledge. notably, this phenomenon disproportionately affects low-resource languages, exacerbating the challenges faced by mmts in achieving true cross-lingual competence. as part of our methodology, we design control tasks @xcite in which, during fine-tuning, the words within the instances have been randomly shuffled, and then, the model is evaluated on the original, i.e., not shuffled, test data. surprisingly, our experiments show that although these new tasks do not provide the model with meaningful linguistic knowledge related to the target task, there is only a slight drop in their cross-lingual performance in both single and twolanguage evaluation settings. these results demonstrate that mmts' understanding tends to be more reliant on surface-level patterns rather than linguistic comprehension. our experiments show from several angles that current mmts' cross-lingual evaluation setups do not give us a clear and faithful picture of their cross-lingual ability. our findings question the extent of high crosslinguality in language models and prompt us to pay more attention to the interpretation of knowledge transfer and cross-lingual ability in multilingual models solely based on their performance on downstream tasks. analyzing mmt. the rise of multilingual models has sparked significant interest to understand their linguistic capabilities across different languages. @xcite have discovered subspaces within mbert representations that can capture syntactic tree distances across different languages. aligned with their finding, numerous studies have further explored the potential of multilingual models to capture both syntactic @xcite @xcite and semantic knowledge @xcite for a wide range of languages. several studies have delved into understanding the factors influencing the cross-lingual ability of multilingual models. @xcite explored the effectiveness of multilingual bert in transferring knowledge across languages using ner and pos tagging tasks, noting the impact of language similarity on performance. @xcite examined cross-linguality from a language structure perspective, emphasizing the significance of the composition property in facilitating cross-lingual transfer. @xcite analyzed representation similarities and discovered a strong connection between hidden cross-lingual similarity and the model's performance on downstream tasks. building upon this finding, @xcite identified a correlation between token embedding alignment and zero-shot transfer across diverse tasks. in the realm of multilingual models, both the design of new models and the examination of existing ones rely heavily on evaluating their performance in zero-shot cross-lingual scenarios. however, there remains the question of how accurately we can interpret their performance and its implications for the models' cross-lingual abilities. to shed light on this issue, our work focuses on investigating the faithfulness of the prevailing methods used to evaluate zero-shot cross-lingual performance in the literature. by doing so, we aim to provide deeper insights into the interpretation of model performance and its relationship to the cross-lingual capabilities of multilingual models. ## cross-lingual evaluation the ability of a multilingual model to effectively generalize across languages on downstream tasks is the key factor in determining its cross-linguality @xcite @xcite . more specifically, a multilingual model is considered cross-lingual if it can successfully perform tasks in languages not seen during task fine-tuning. however, relying solely on the ultimate performance of multilingual models on a specific target task has significant drawbacks, as it can potentially create misconceptions about their true crosslingual abilities. this evaluation approach lacks clarity in distinguishing between the extent of crosslingual knowledge transfer and surface-level and non-linguistic features. it is possible for a multilingual model to achieve high performance on a task without possessing a deep semantic understanding of a language, instead mostly relying on languageindependent and shallow knowledge, as previously reported for monolingual language models @xcite @xcite . it is worth mentioning that fully disentangling linguistic and shallow (task-specific) knowledge is almost impossible for most nlp tasks. however, trying to separate these types of knowledge as much as possible gives us a clearer and more accurate perspective regarding the linguistic knowledge captured by models across languages. in the following parts, we employ three different target tasks, namely multilingual natural language inference, paraphrase identification, and question answering, to study the cross-lingual ability of multilingual models. we utilize multilingual bert @xcite and the base version of xlm-r @xcite for our experiments as they are among the most widely used multilingual models. we also employ infoxlm @xcite trained on a cross-lingual objective and parallel data to investigate the role of explicit pre-trained cross-lingual objective on the cross-lingual ability. foot_1 ## breakdown analysis in this section, we delve into the reasons behind the lack of success exhibited by multilingual models in the across language setup. our analysis begins by exploring the individual contributions of each class to the overall tasks' performance. by doing so, we aim to uncover the specific instances that present difficulties for multilingual models and subsequently lead to a decline in their performance. ## control tasks in this section, we borrow the idea of control tasks in probing and employ them to assess the cross-lingual abilities of mmts. these tasks are indeed (partially) random sequences, allowing us to evaluate the models' performance when it does not receive linguistic cues. this unconventional approach provides valuable insights into the extent of the meaningful numerical performance of multilingual models on the current multilingual benchmarks. to this aim, we randomly shuffle the inputs and fine-tune the models on these shuffled instances. more specifically, for the nli and pi tasks, we shuffle the words within every sentence, and for the qa task, we shuffle the question words and keep the original context. it is evident that the designed tasks are only marginally related to human language comprehension. however, surprisingly, the performance of the models on these tasks, as shown in in the case of the nli task, we observe a slight 4% drop in accuracy on the test sets when the model is fine-tuned on a nonsensical task. similarly, the qa model exhibits a drop of approximately 3%. these findings suggest that the current test sets may not provide sufficient quality for effectively evaluating the cross-lingual capabilities of multilingual models. out of all the tasks, the pi task stands out with its random performance, indicating that the fine-tuned model with shuffled data lacks the necessary knowledge to complete the task successfully. this aligns with our expectations from a high-quality dataset. ## future directions in this work, we have demonstrated that current performance-based methods for evaluating crosslingual abilities are insufficient. the main concern now is how to assess the cross-lingual capabilities of multilingual models. firstly, we should be aware of the shortcomings of existing multilingual benchmarks commonly used for evaluating cross-lingual knowledge transfer. as we described in our study, current tasks and datasets are prone to artifacts leading to exaggerated high performance. in addition to the random baseline, which is defined based on the number of labels for classification tasks, we suggest having a secondary baseline for every task. the secondary baseline could be considered the performance of a simple neural network on the task or the performance on the control tasks. we leave the exploration of this avenue to future work. another alternative could be our suggested setup, the across language approach, which involves multiple languages. the proposed evaluation setup is not only valuable for theoretical analysis but also reflects real-world scenarios where nlp systems need to integrate knowledge from different language sources. ## conclusion in this paper, we take a fresh perspective on the cross-lingual ability of multilingual models. through comprehensive experiments, we explored their capacity to simultaneously leverage knowledge from multiple languages. our findings show that multilingual models struggle to establish connections between knowledge spaces across languages, resulting in subpar performance on crosslanguage task setups, i.e., when there are multiple languages in the input. our results show that the previously reported high performance in the zeroshot setting predominantly stems from the transfer of shallow, language-independent knowledge. surprisingly, we observed that dataset artifacts, rather than intrinsic linguistic features, are predominantly transferred across languages. this challenges the notion of relying solely on multilingual models' performance for assessing their true cross-lingual capabilities. to assess the quality of existing multilingual benchmarks, we conducted fine-tuning experiments on control tasks with nonsensical input. surprisingly, even when fine-tuned on meaningless tasks, the models demonstrated exceptionally high performance, prompting concerns regarding the quality of current multilingual datasets. in light of these insights, in the future, we plan to explore novel task and data-independent approaches to gain a more accurate understanding of multilingual models' true cross-lingual abilities.