Exploring the Relationship between Alignment and Cross-lingual Transfer in Multilingual Transformers (2023)

Abstract: Without any explicit cross-lingual training data, multilingual language models can achieve cross-lingual transfer. One common way to improve this transfer is to perform realignment steps before fine-tuning, i.e., to train the model to build similar representations for pairs of words from translated sentences. But such realignment methods were found to not always improve results across languages and tasks, which raises the question of whether aligned representations are truly beneficial for cross-lingual transfer. We provide evidence that alignment is actually significantly correlated with cross-lingual transfer across languages, models and random seeds. We show that fine-tuning can have a significant impact on alignment, depending mainly on the downstream task and the model. Finally, we show that realignment can, in some instances, improve cross-lingual transfer, and we identify conditions in which realignment methods provide significant improvements. Namely, we find that realignment works better on tasks for which alignment is correlated with cross-lingual transfer when generalizing to a distant language and with smaller models, as well as when using a bilingual dictionary rather than FastAlign to extract realignment pairs. For example, for POS-tagging, between English and Arabic, realignment can bring a +15.8 accuracy improvement on distilmBERT, even outperforming XLM-R Large by 1.7. We thus advocate for further research on realignment methods for smaller multilingual models as an alternative to scaling.

## introduction with the more general aim of improving the understanding of multilingual large language models (mllm), we study the link between the multilingual alignment of their representations and their ability to perform cross-lingual transfer learning, and investigate conditions for realignment methods to improve cross-lingual transfer. figure 1 : cross-lingual transfer between english and arabic with and without realignment, using a bilingual dictionary. for some tasks, realignment can make small models competitive with a large baseline. but the ability of mllms to generalize across languages is highly correlated with the similarity between the training language (often english) and the language to which we hope to transfer knowledge @xcite . for distant and low-resources languages, ctl with mbert can give worse results than fine-tuning a transformer from scratch @xcite . realignment methods @xcite , sometimes called adjustment or explicit alignment, aim to improve the cross-lingual properties of an mllm by trying to make similar words from dif-ferent languages have closer representations. realignment methods typically require a translation dataset and an alignment tool, like fastalign @xcite , to extract contextualized pairs of translated words that will be realigned. despite some encouraging results on specific tasks, current realignment methods might not consistently improve cross-lingual zero-shot abilities of mbert and xlm-r @xcite . when tested with several seeds on various finetuning tasks, improvements brought by realignment are not always significant and do not compare with the gain brought by scaling the model, e.g., from xlm-r base to xlm-r large. however, these realignment methods were not tried on smaller models like distilmbert as we do here. the mitigated results of realignment methods raise the question of whether cross-lingual transfer is at all linked with multilingual alignment. if improving alignment does not necessarily improve ctl, then the two might not be correlated. despite the ability of mbert and xlm-r to perform ctl, there lacks consensus on whether they actually hold aligned representations @xcite . we thus investigate the link between alignment and ctl, with three contributions: (1) we find a high correlation between multilingual alignment and cross-lingual transfer for multilingual transformers, (2) we show that, depending on the downstream task, fine-tuning on english can harm the alignment to different degrees, potentially harming cross-lingual transfer, and (3) we link our findings to realignment methods and identify conditions under which they seem to bring the most significant improvements to zero-shot transfer learning, particularly on smaller models as shown on fig. 1 . ## related work current realignment methods are applied on a pretrained model before fine-tuning in one language (typically english). common tasks are natural language inference (nli), named entity recognition (ner), part-of-speech tagging (pos-tagging) or question answering (qa). the model is then expected to generalize better to other languages for the task than without the realignment. realignment methods rely on pairs of words extracted from translated sentences using a word alignment tool, usually fastalign @xcite , but other tools like awesome-align @xcite could be used. various realignment objectives are used to bring closer together the contextualized embeddings of words in such pairs: using a linear mapping @xcite , a ℓ 2 -based loss with regularization to avoid degenerative solutions @xcite , or a contrastive loss @xcite . existing realignment methods might not significantly improve cross-lingual transfer. despite improvements on nli @xcite @xcite or on dependency parsing @xcite , the results might not hold across tasks and languages. a comparative study by @xcite showed that methods based on linear mapping are effective only on "moderately close languages", whereas ℓ 2 -based loss improves results for "extremely distant languages". this latter ℓ 2 -loss was shown to work well on a nli task, but not for all languages on a ner task, and to be even detrimental for qa tasks @xcite . finally, wu and dredze (2020b) compared linear mapping realignment, ℓ 2 -based realignment and contrastive learning on several tasks, languages and models, performing several runs. they found that existing methods do not bring consistent improvements over no realignment. expecting realignment methods to succeed implies a direct link between the multilingual alignment of the representations produced by a model and its ability to perform ctl. however, there isn't any strong consensus on whether multilingual transformers have well-aligned representations @xcite , let alone on whether better-aligned representations lead to better ctl. assessing the multilingual alignment of contextualized representations can take many forms. pairs of words are extracted from translated sentences, usually with fastalign or a bilingual dictionary @xcite . then, after building contextualized representations of the words of each pair, the distribution of their similarity can be compared with that of random pairs of words @xcite . but this method can lead to incorrect conclusions @xcite . a high overlap in the distribution of similarities between related and random pairs means that sometimes random pairs can have higher similarities than related pairs. but since those pairs do not necessarily involve the same words, a high overlap does not mean that any word is closer to an unrelated one than to a related one. an alternative is to compare a related pair to its neighbors @xcite , which shows that realignment methods indeed improve multilingual alignment and that fine-tuning can harm this alignment. another similar approach consists in designing a nearest-neighbor search criterion. this was done for sentence-level representations @xcite and for word-level alignment @xcite , showing that mllms like mbert have a multilingual alignment that is competitive with static embeddings @xcite explicitly aligned with a supervised alignment method @xcite . ## method to study the link between multilingual alignment and cross-lingual transfer (ctl), we need a way to evaluate alignment and ctl. we use a relative difference to evaluate ctl, we discuss different methods for evaluating alignment, and describe the realignment method used in our experiments. ## correlation between alignment and ctl we measure the correlation between multilingual alignment and cross-lingual transfer (ctl) across models, languages and seeds. we also compare the correlation between alignment before fine-tuning and after fine-tuning with ctl and with different alignment measures. spearman's rank correlation is measured between alignment before or after fine-tuning and ctl. the english-target alignment is computed for each target language with the method described in section 3.2 and is compared with the transfer ability from english to that same target language with the metric described in section 3.1. results show that strong alignment is better correlated to cross-lingual transfer than weak alignment. with the exception of two tasks after finetuning (ner and nli), strong alignment has a marginally higher correlation with ctl. this is particularly noticeable when looking at alignment before fine-tuning on the last layer, going from a correlation between 0.51 and 0.69 for weak alignment to one ranging from 0.82 to 0.86 for strong alignment. tab. 2 also shows that for nli, the alignment on the penultimate layer seems better correlated to cross-lingual transfer than with the last layer. a relatively important gap in correlation is measured between the last and the second-before-last layer for all cases except for strong alignment before fine-tuning. the fact that alignment on the penultimate layer would correlate better than the last for nli can be explained by the sentence-level nature of the task. for sentence classification tasks, the classification head reads only the representation of the first token of the last layer, which is computed from the representations of all the tokens at the previous layer, leading to a pooling of the penultimate layer. despite the different values observed, there seems to be no significant difference between correlation for alignment measured before and after fine-tuning, and a careful analysis of confidence interval obtained with bootstrapping @xcite can confirm this (cf. appendix c for detailed results). fig. 2 shows the relation between ctl and english-target strong alignment measured after fine-tuning measured in four situations to further illustrate the link between alignment and transfer. fig. 2b shows one of the cases with higher correlation (0.92). the correlation seems to hold well across models (forms) and languages (colors). however, for a given model and language, the random seed for fine-tuning seems to be detrimental to the correlation, although at a small scale. hence, alignment might not be the only factor to affect cross-lingual generalization as the model initializa-tion or the data shuffling seems to play a smaller role. fig. 3 shows a case with one of the lowest correlations between strong alignment and ctl (0.70). it seems that models and initialization seeds have a higher impact on alignment than on ctl. for example, in the case of english-french alignment (green), ctl is between 0.0 and -0.1, whatever the model and seed, not overlapping with other target-english language pairs, but alignment varies between approximately 0.05 and 0.5, overlapping with all other language pairs. interestingly, the penultimate layer has a higher accuracy (0.82), suggesting that for ner the last layer is not necessarily the one for which alignment correlates the most with ctl. for two of the three tested tasks (ner and postagging), it must be noted that the ctl metric is not strictly comparable across languages since the test sets for each language are of different domains and sizes (cf. section 3.4). however, for the third task (nli), each test set is a translation of the english one, and thus the ctl metric is strictly comparable in that case. this might explain why correlations are higher for the nli task than the others. nevertheless, the observed correlation for the two other tasks is still significantly high, which suggests that the general tendency might not be affected by the differences in domains and sizes in the test sets. ## the impact of fine-tuning on alignment to study the link between alignment and crosslingual transfer (ctl), we also look at the impact of fine-tuning over alignment. we've already shown that strong alignment is highly correlated with ctl. however, we weren't able to conclude whether alignment measured before or after finetuning was better correlated to ctl abilities. to understand the difference between both measures, we study in this section the impact of fine-tuning on the alignment of mllms representations. we use the same fine-tuning runs as in the previous section (4). tab. 3 shows the relative variation in alignment before and after fine-tuning for all tasks and models tested and for three languages for clarity (complete results in appendix d). the relative difference is built in the same way as the cross-lingual transfer evaluation (eq. 1). negative values indicate a drop in alignment. alignment is measured at the last layer. fig. 4 and 5 show a breakdown by layer for for certain combinations of models and tasks, fine-tuning is detrimental to multilingual alignment. distilmbert and mbert mainly show a decrease in alignment for pos-tagging and ner, and smaller improvements than other models on nli. however, pos-tagging is the only of the three tasks which shows dramatic drops where alignment can be reduced by as much as 96%. the drop in alignment can be explained by catastrophic forgetting. if the model is only trained on a monolingual task, it might not retain information about other languages or about the link between english and other languages. what is more surprising is the increase in alignment obtained in other cases. xlm-r base and large, which are larger models than mbert and distilmbert, have a relative increase that can go as high as 25.36 on the nli task for distant languages. and although these increases are from a small alignment measure, we still observe a large increase for middle layers where the initial alignment is already quite high (cf. fig. 5 ). the alignment of larger models being less harmed by fine-tuning is coherent with the fact that those same larger models have been shown to have better ctl abilities. fig. 4 shows that more layers seem to mitigate the potentially negative impact of fine-tuning on alignment, as it affects mainly the layers closest to the last one and as the initial alignment measure is globally higher for xlm-r than distilmbert (before fine-tuning: ≈0.25 against ≈0.008). giving a definitive answer as to why different tasks have different impacts on alignment might need further research. but one could already argue that each task corresponds to different levels of abstraction in nlp. tasks with a low level of abstraction like pos-tagging might rely on the word form itself and thus on more language-specific components of the representations, which when enhanced, decreases alignment. on the other hand, nli has a higher level of abstraction, requiring the meaning rather than the word form, which might be encoded in deeper layers @xcite which are more aligned. fine-tuning mllms on a downstream task has an impact on the multilingual alignment of the representations produced by the model. for "smaller" language models, it is systematically detrimental, as well as for certain tasks like pos-tagging. this might explain why some realignment methods might not work for all models nor all tasks @xcite . ## impact of realignment on cross-lingual transfer we have already shown that the correlation between multilingual alignment and cross-lingual transfer (ctl) is high (section 4). but we do not know whether they are more directly linked. in this section, we try to identify the conditions under which improving alignment in multilingual models leads to improvement in ctl. sequential realignment is the usual way to perform realignment: realignment steps are performed on the pre-trained model before fine-tuning. we propose to compare it with joint alignment, where we optimize simultaneously for the realignment and the downstream task (more details in appendix a), to try and identify whether alignment before or after fine-tuning is more strongly related to ctl. in the same settings as the previous experiments (tasks, models and languages, and number of seeds), we fine-tune models in english with different realignment methods and evaluate ctl on different languages. following a similar setting as @xcite , realignment data from the five pairs of languages (english-target) is interleaved to form a single multilingual realignment dataset. models are fine-tuned on pos-tagging or ner for five epochs and 2 epochs for nli because its training data is larger. we use the opus100 translation dataset @xcite ) from which we extract pairs of words using bilingual dictionaries. we also tested with the multiun translation data @xcite , which conditioned our choice of languages, and with other ways to extract alignment pairs: fastalign @xcite pos ner nli distilmbert 73.1 57.4 64.2 + before 78.0 58.9 64.3 + joint 77.9 59.6 65.0 mbert 74.3 62.2 69.7 + before 78.2 62.7 68.7 + joint 78.3 64.8 70.0 xlm-r base 78.8 60.4 74.0 + before 79.9 63.9 72.7 + joint 79.4 63.0 74.6 xlm-r large 79.6 65.0 80.0 condensed results are reported on tab. 4, averaged on the five languages. a breakdown by languages for the pos-tagging task and two models is shown on tab. 5. it shows that realignment methods improve performance only on certain tasks, models and language pairs. realignment methods, either sequential or joint, provide significant improvement for all models for the pos-tagging task, but less significant ones for ner, and no significant improvement for nli. the positive impact of realignment on cross-lingual transfer seems to be mirrored by the negative impact of fine-tuning over alignment. indeed, postagging is also the task for which fine-tuning is the most detrimental to multilingual alignment, as shown in the previous section. the same parallel can be drawn for models. dis-tilmbert is the model that benefits the most from realignment. it is also the one whose alignment suffers the most from fine-tuning. smaller multi-lingual models seem to benefit more from realignment, as well as they see their multilingual alignment reduced after fine-tuning. in the same way that fine-tuning mainly affects the deeper layers, it is possible that realignment might affect only those deeper layers. this would mean that most layers would have their alignment significantly improved for small models like distilmbert (6 layers), while larger models might be only superficially realigned. finally, besides tasks and models, it can also be observed that the impact of realignment varies across language pairs (tab. 5). although we did not test on many language pairs, results are coherent with the idea that realignment methods tend to work better on distant pairs of languages @xcite . on a side note, our controlled experiment does not allow us to conclude whether it is more important to improve alignment before fine-tuning or after. it seems that alignment measured before and after fine-tuning are equally important to crosslingual transfer. realignment methods unsurprisingly provide better results when the alignment is lower, be it before or after fine-tuning. distant languages and small models have lower alignment, and pos-tagging is a task where alignment decreases after fine-tuning. realignment helps only up to a certain point where representations are already well aligned, and ctl gives already good results. for distilmbert on pos-tagging for transfer from english to arabic, it provides a +15.8 improvement over baseline, even outperforming xlm-r large by 1.7 points. in such conditions, realignment is an interesting alternative to scaling for multilingual models. if realignment succeeds in some favorable conditions, then how can we explain that realignment methods were shown to not be significantly improving ctl on several tasks, including pos-tagging @xcite ? firstly, to the best of our knowledge, realignment was never tried on distilmbert or other models of equivalent size. secondly, tab. 6 shows that it might be partly due to an element of the realignment methods that was overlooked: the source of related pairs of words. the way pairs are extracted seems to be crucial to the success of realignment methods. tab. 6 shows the effect of different types of pairs extraction in realignment methods. realignment methods using pairs extracted with fastalign or pos ner xlm-r base 78.8 60.4 + before fastalign 78.6 61.4 + before awesome 78.6 62.0 + before dico 79.9 63.9 + joint fastalign 78.0 62.1 + joint awesome 77.8 62.3 + joint dico 79.4 63.0 awesome-align do not provide significant improvements over the baseline, whereas using a bilingual dictionary does. using a bilingual dictionary might be more accurate for extracting translated pairs @xcite . another explanation could be that the type of words contained in a dictionary might help since it might contain more lexical words holding meaning and fewer grammatical words. ## conclusion we have shown that multilingual alignment, measured using a nearest-neighbor search among translated pairs of contextualized words, is highly correlated with the cross-lingual transfer abilities of multilingual models (or at least multilingual transformers). strong alignment was also revealed to be better correlated to cross-lingual transfer than weak alignment. then we investigated the impact of fine-tuning (necessary for cross-lingual transfer) on alignment as well as the impact of realignment methods on cross-lingual transfer. fine-tuning was revealed to have a very different impact on alignment depending on the downstream task and the model. where lower-level tasks seemed to have the most impact and smaller models seemed to be the most affected. conversely, realignment methods were shown to work better on those same tasks and models. ultimately, realignment works unsurprisingly better when the baseline alignment (before or after fine-tuning) is lower. we also showed that using a bilingual dictionary for extracting pairs for realignment methods improves over the commonly used fastalign and over a more precise neural aligner (awesome-align). it's worth noting that realignment works particularly well for a small model like distilmbert (66m parameters), allowing it in some cases to ob-tain competitive results with xlm-r large (354m parameters). this advocates for further research on realignment for small transformers to build more compute-efficient multilingual models. finally, further research is needed to investigate additional questions, like whether cross-lingual transfer is more directly linked to alignment before or after fine-tuning, or to alignment at certain layers for certain tasks. to answer these questions, more large-scale experiments could be performed on more tasks and especially on more languages to obtain correlation values with smaller confidence intervals.