m PLM -Sim: Better Cross-Lingual Similarity and Transfer in Multilingual Pretrained Language Models (2024)

Abstract: Recent multilingual pretrained language models (mPLMs) have been shown to encode strong language-specific signals, which are not explicitly provided during pretraining. It remains an open question whether it is feasible to employ mPLMs to measure language similarity, and subsequently use the similarity results to select source languages for boosting cross-lingual transfer. To investigate this, we propose mPLM-Sim, a language similarity measure that induces the similarities across languages from mPLMs using multi-parallel corpora. Our study shows that mPLM-Sim exhibits moderately high correlations with linguistic similarity measures, such as lexicostatistics, genealogical language family, and geographical sprachbund. We also conduct a case study on languages with low correlation and observe that mPLM-Sim yields more accurate similarity results. Additionally, we find that similarity results vary across different mPLMs and different layers within an mPLM. We further investigate whether mPLM-Sim is effective for zero-shot cross-lingual transfer by conducting experiments on both low-level syntactic tasks and high-level semantic tasks. The experimental results demonstrate that mPLM-Sim is capable of selecting better source languages than linguistic measures, resulting in a 1%-2% improvement in zero-shot cross-lingual transfer performance.

## introduction recent multilingual pretrained language models (mplms) trained with massive data, e.g., mbert @xcite , xlm-r @xcite and bloom @xcite , have become a standard for multilingual representation learning. follow-up works @xcite @xcite show that these mplms encode strong languagespecific signals which are not explicitly provided during pretraining. however, the possibility of using mplms to measure language similarity and utilizing the similarity results to pick source languages for enhancing cross-lingual transfer is not yet thoroughly investigated. to investigate language similarity in mplms, we propose mplm-sim, a measure that leverages mplms and multi-parallel corpora to measure similarity between languages. using mplm-sim, we intend to answer the following research questions. we compute pearson correlation between similarity results of mplm-sim and linguistic similarity measures. the results show that mplm-sim has a moderately high correlation with some linguistic measures, such as lexical-based and languagefamily-based measures. additional case studies on languages with low correlation demonstrate that mplms can acquire the similarity patterns among languages through pretraining on massive data. we make a comprehensive comparison among a diverse set of 11 mplms in terms of architecture, modality, model size, and tokenizer. the experimental results show that input modality (text or speech), model size, and data used for pretraining have large effects on mplm-sim while tokenizers and training objectives have little effect. previous works @xcite @xcite @xcite have shown that the performance of cross-lingual transfer positively correlates with linguistic similarity. however, we find that there can be a mismatch between mplm subspaces and linguistic clusters, which may lead to a failure of zero-shot cross-lingual transfer for lowresource languages. intuitively, mplm-sim can select the source languages that boost cross-lingual transfer better than linguistic similarity since it captures the subspaces learned during pretraining (and which are the basis for successful transfer). to examine this, we conduct experiments on four datasets that require reasoning about different levels of syntax and semantics for a diverse set of lowresource languages. the results show that mplm-sim achieves 1%-2% improvement over linguistic similarity measures for cross-lingual transfer. for each sentence x in the multi-parallel corpus, the mplm computes its sentence embedding for the ith layer of the mplm: h @xmath0. for mplms with bidirectional encoders, including encoder architecture, e.g., xlm-r, and encoderdecoder architecture, e.g., mt5, e(•) is a mean 2 monolingual corpora covering multiple languages can be also used to measure language similarity. our initial experiments ( §b.1) show that parallel corpora yield better results while using fewer sentences than monolingual corpora. therefore, we use parallel corpora for our investigation. for ith layer, the similarity of each language pair is computed using the sentence embeddings of all multi-parallel sentences. specifically, we get the cosine similarity of each parallel sentence of the language pair, and then average all similarity scores across sentences as the final score of the pair. finally, we have a similarity matrix s @xmath1. ## conclusion in this paper, we introduce mplm-sim, a novel approach for measuring language similarities. extensive experiments substantiate the superior performance of mplm-sim compared to linguistic similarity measures. our study reveals variations in similarity results across different mplms and layers within an mplm. furthermore, our findings reveal that mplm-sim effectively identifies the source language to enhance cross-lingual transfer. the results obtained from mplm-sim have significant implications for multilinguality. on the one hand, it can be further used in linguistic study and downstream applications, such as cross-lingual transfer, as elaborated in the paper. on the other hand, these findings provide valuable insights for improving mplms, offering opportunities for their further development and enhancement.