Identifying the Correlation Between Language Distance and Cross-Lingual Transfer in a Multilingual Representation Space (2023)

Abstract: Prior research has investigated the impact of various linguistic features on cross-lingual transfer performance. In this study, we investigate the manner in which this effect can be mapped onto the representation space. While past studies have focused on the impact on cross-lingual alignment in multilingual language models during fine-tuning, this study examines the absolute evolution of the respective language representation spaces produced by MLLMs. We place a specific emphasis on the role of linguistic characteristics and investigate their inter-correlation with the impact on representation spaces and cross-lingual transfer performance. Additionally, this paper provides preliminary evidence of how these findings can be leveraged to enhance transfer to linguistically distant languages.

## introduction it has been shown that language models implicitly encode linguistic knowledge @xcite . in the case of multilingual language models (mllms), previous research has also extensively investigated the influence of these linguistic features on cross-lingual transfer performance @xcite @xcite . however, limited attention has been paid to the impact of these factors on the language representation spaces of mllms. despite the fact that state-of-the-art mllms such as mbert @xcite and xlm-r @xcite , use a shared vocabulary and are intended to project text from any language into a language-agnostic embedding space, empirical evidence has demonstrated that these models encode language-specific information across all layers @xcite . this leads to the possibility of identifying distinct monolingual representation spaces within the shared multilingual representation space @xcite . past research has focused on the cross-linguality of mllms during fine-tuning, specifically looking at the alignment of representation spaces of different language pairs @xcite . our focus, instead, is directed towards the absolute impact on the representation space of each language individually, rather than the relative impact on the representation space of a language compared to another one. isolating the impact for each language enables a more in-depth study of the inner modifications that occur within mllms during fine-tuning. the main objective of our study is to examine the role of linguistic features in this context, as previous research has shown their impact on cross-lingual transfer performance. more specifically, we examine the relationship between the impact on the representation space of a target language after fine-tuning on a source language and five different language distance metrics. we have observed such relationships across all layers with a trend of stronger correlations in the deeper layers of the mllm and significant differences between language distance metrics. additionally, we observe an inter-correlation among language distance, impact on the representation space and transfer performance. based on this observation, we propose a hypothesis that may assist in enhancing cross-lingual transfer to linguistically distant languages and provide preliminary evidence to suggest that further investigation of our hypothesis is merited. ## related work in monolingual settings, @xcite found that, after pre-training, bert encodes different linguistic features in different layers. @xcite showed that language models do not forget these linguistic structures during fine-tuning on a downstream task. conversely, @xcite have shown that during fine-tuning in multilingual settings, mbert forgets some languagespecific information, resulting in a more crosslingual model. at the representation space level, @xcite and @xcite studied the impact of fine-tuning on mbert's cross-linguality layer-wise. however, their research was limited to the evaluation of the impact on cross-lingual alignment comparing the representation space of one language to another, rather than assessing the evolution of a language's representation space in isolation. ## correlation analysis relationship between the impact on the representation space and language distance. given the layer-wise differences of mbert's crosslinguality @xcite , we measure the correlation between the impact on the representation space and the language distances across all layers. figure 1 shows almost no significant correlation between representation space impact and inventory or phonological distance. geographic and syntactic distance mostly show significant correlation values at the last layers. only the genetic distance correlates significantly across all layers with the impact on the representation space. relationship between language distance and cross-lingual transfer performance. ## conclusion in previous research, the effect of fine-tuning on a language representation space was usually studied in relative terms, for instance by comparing the cross-lingual alignment between two monolingual representation spaces before and after fine-tuning. our research, however, focused on the absolute impact on the language-specific representation spaces within the multilingual space and explored the relationship between this impact and language distance. our findings suggest that there is an intercorrelation between language distance, impact on the representation space, and transfer performance which varies across layers. based on this finding, we hypothesize that selectively freezing layers during fine-tuning, at which specific inter-correlations are observed, may help to reduce the transfer performance gap to distant languages. although our hypothesis is only supported by three pilot experiments, we anticipate that it may stimulate further research to include an assessment of our hypothesis.