A lign F reeze: Navigating the Impact of Realignment on the Layers of Multilingual Models Across Diverse Languages (2025)

Abstract: Realignment techniques are often employed to enhance cross-lingual transfer in multilingual language models, still, they can sometimes degrade performance in languages that differ significantly from the fine-tuned source language. This paper introduces AlignFreeze, a method that freezes either the layers’ lower half or upper half during realignment. Through controlled experiments on 4 tasks, 3 models, and in 35 languages, we find that realignment affects all the layers but can be the most detrimental to the lower ones. Freezing the lower layers can prevent performance degradation. Particularly, AlignFreeze improves Part-of-Speech (PoS) tagging performances in languages where full realignment fails: with XLM-R, it provides improvements of more than one standard deviation in accuracy in seven more languages than full realignment.

## introduction multilingual language models (mlms) like xlm-r @xcite or mbert @xcite can perform cross-lingual transfer @xcite . once finetuned on a specific task in english, these models perform well on that same task when evaluated in other languages. while this can be useful for languages where fine-tuning data might be missing, cross-lingual transfer is often less efficient for languages that differ greatly from english @xcite , which unfortunately are the languages that would benefit the most from such ability. with an approach similar to building multilingual word embeddings @xcite @xcite , realignment explicitly re-trains an mlm for multilingual alignment with the hope of improving its cross-lingual transfer abilities. while some work report some level of success @xcite @xcite , systematic evaluations show that realignment does not consistently improve cross-lingual transfer abilities and can significantly degrade them in some cases @xcite . the relative failure of realignment raises the question of whether better multilingual alignment necessarily implies stronger cross-lingual transfer abilities. previous work has found that mlms have good multilingual alignment, on top of their crosslingual transfer abilities @xcite , and there even seems to be a strong link between alignment and cross-lingual transfer @xcite , although the correlation is not causation and it remains that realignment often fails. if better alignment is linked to better crosslingual transfer, we hypothesize that realignment has some adverse effect that induces catastrophic forgetting of other important features of the model. to better understand this side-effect of realignment and how the different layers are affected, we propose alignfreeze. in this method, half of the model layers are frozen during realignment. with a simple controlled experiment, we compare the impact on the lower and the upper layers. we find that realignment impacts all layers, but is particularly detrimental on lower layers, namely for a low-level task like pos tagging. ## background on realignment realignment explicitly enforces the multilingual alignment of embeddings produced by multilingual models. it trains a multilingual model to produce similar representations for corresponding words in translated sentences. two resources are needed: a translation dataset and a word alignment tool which, in our experiments, is either fastalign @xcite , @xcite , or a simple look-up table based on bilingual dictionaries @xcite as proposed in @xcite . in our experiments, we use the realignment method proposed by @xcite , where a contrastive loss maximizes the similarity between the representations of a pair of corresponding words (h and aligned(h)) compared to all other possible pairs of words in a batch (h of size b) of pairs of translated sentences: t is the temperature, a hyperparameter set to 0.1. ## methodology we introduce alignfreeze, a realignment method that relies on partial freezing to preserve half of the weights of an mlm during realignment. because full realignment was shown not to work consistently (wu and dredze, 2020), we hypothesize that applying realignment on the whole model could trigger some catastrophic forgetting of information useful to downstream cross-lingual tasks. to help mitigate that and better understand the impact of realignment, alignfreeze freezes half of the layers of the mlm during realignment only. freezing strategies for the sake of simplicity and to reduce the number of experimental runs, we work with only two freezing strategies: 1) frontfreezing, which freezes the lower-half layers while the remaining layers are realigned; and 2) backfreezing, which freezes upper-half layers instead. assuming that basic linguistic features are encoded in the lower layers while the top ones retain higher-level information @xcite , frontfreezing aims to preserve the foundational language understanding captured in the early layers while enabling task-specific adaptation in the later layers. back-freezing seeks to maintain the abstract, high-level representations developed in the deeper layers while fine-tuning the model's basic linguistic features. our approach intentionally employs a straightforward freezing strategy, not to establish a new state-of-the-art realignment method, but to better understand the conditions under which realignment fails and how to mitigate its failure. the freezing is applied only during realignment. thus, alignfreeze can be described with the following steps: 1) take a multilingual language model (mlm), 2) freeze half of its layers, 3) train the remaining weights for the realignment loss, 4) unfreeze the frozen layers, 5) perform fine-tuning on the whole model for cross-lingual transfer. datasets realignment dataset: we use the opus-100 dataset @xcite for the realignment phase. opus-100 is a multilingual parallel corpus that includes sentence pairs across multiple languages. downstream task dataset: we evaluate multilingual models on three tasks: pos tagging, named entity recognition (ner), natural language inference (nli), and question answering (qa). for pos tagging, we use the universal dependencies dataset @xcite , which provides annotated treebanks for a wide range of languages. for ner, we use the wikiann dataset @xcite . for nli, we use the cross-lingual natural language inference (xnli) corpus @xcite . for qa, we use the xquad dataset @xcite . following gaschi et al. (2023), we work with three models: distilmbert (sanh et al., 2019), mbert (devlin et al., 2019), and xlm-r base (conneau et al., 2020). distilmbert is a smaller version of mbert (devlin et al., 2019) obtained through distillation (sanh et al., 2019). distilmbert, mbert, and xlm-r are all transformer-based masked multilingual models. languages we use english as the source language for fine-tuning. we evaluate on 34 languages for pos-tagging and ner" 12 for nli, and 11 for qa. for realignment, we use the 34 available languages for pos tagging, ner" nli, and qa. using the same setting allows for comparison of results across tasks and also improves the outcome (cf. appendix c.2). we use all the languages that our resources allow: every language must be present in the translation dataset, the bilingual dictionaries, and one of the downstream datasets. the full list can be found in the subsection b.1. further details about the implementation can be found in appendix b and in the source code 1 . ## results and discussion finding 1: full realignment fails in many cases. as already observed by previous work @xcite @xcite , full realignment isn't always successful. finding 2: alignfreeze (front) mitigates some of the failures of realignment. freezing the lower layers during realignment often improves results for cases where full realignment fails. front-freezing can mitigate some failure cases of full realignment, thus realignment can have a detrimental effect on the lower layers. on the other hand, back-freezing seems to have a less important impact on realignment. ## conclusion this study introduces alignfreeze, a method using partial freezing to improve cross-lingual transfer in multilingual language models. our experiments demonstrate that alignfreeze effectively mitigates the failure cases of partial realignment by preserving pre-trained knowledge in the lower layers. when it comes to cross-lingual transfer, there does not seem to be any "silver bullet" @xcite ) method that works for all languages, models, and tasks. like realignment itself, and other cross-lingual approaches, align-freeze can help for some situations but not others. alignfreeze can at least be useful for crosslingual pos-tagging with xlm-r. alignfreeze helps better understand how realignment works. it impacts all layers and can be most detrimental to the lower ones, which is more visible on low-level tasks like pos-tagging, that might be encoded in lower layers @xcite . realignment probably fails simply because it is applied to the whole model without hindrance, which explains alignfreeze relative success but also the results of other methods based on adapters like mad-x (pfeiffer et al., 2020).