Understanding Cross-Lingual A lignment— A Survey (2024)

Abstract: Cross-lingual alignment, the meaningful similarity of representations across languages in multilingual language models, has been an active field of research in recent years. We survey the literature of techniques to improve cross-lingual alignment, providing a taxonomy of methods and summarising insights from throughout the field. We present different understandings of cross-lingual alignment and their limitations. We provide a qualitative summary of results from a number of surveyed papers. Finally, we discuss how these insights may be applied not only to encoder models, where this topic has been heavily studied, but also to encoder-decoder or even decoder-only models, and argue that an effective trade-off between language-neutral and language-specific information is key.

## introduction zero-shot cross-lingual transfer using highly multilingual models has been an active subset of multilingual nlp research. in tasks like sentence classification, sequence labelling, or sentence retrieval, all of which rely on encoder representations, crosslingual alignment of those representations is an underlying assumption for zero-shot cross-lingual transfer. once the model has learned to do, e.g., a classification task in a source language, then if the representation of a target language is "aligned" to that of the source language, the model should also be able to classify target language items. as we define it, cross-lingual alignment means that words or sentences with similar semantics: 1) are more similar in the representation space than words or sentences with dissimilar semantics. this way of looking at alignment can be defined in "weak" or "strong" terms (see § 2). 2) allow a prediction head trained on a source language to recognise the relevant patterns in the target language. we argue that this is related to the first view but, importantly, is more nuanced. this allows us to discuss the literature in a new way. these two criteria are not guaranteed to be fulfilled through unsupervised pre-training, thus motivating various efforts to improve cross-lingual alignment. we survey important papers in this area between 2019 and 2023. these works propose new training objectives, new pre-trained models, contrastive fine-tuning, or post-hoc adjustments of the embedding space. the vast majority of the methods were developed for and applied to multilingual encoder models, chiefly xlm-r and mbert. ## taxonomy of alignment strategies we report on methods for improving zero-shot transfer and increasing cross-lingual alignment. ## what we do and don't know in this section, we discuss broad findings both from the alignment methods summarised above and from recent related analysis papers. future work in this area should follow from open questions. a selection of task results achieved by the alignment methods can be found in appendix b. additionally, appendix c shows which authors provide code or model downloads for reproducibility. contrastive training works. contrastive training is effective for cross-lingual transfer (as well as for other problems), presumably because it reduces the hubness problem and forces models to differentiate representations. it is especially popular for retrieval-based tasks and sentence-level models. pre-training is not everything. whether a model is newly-trained or modified from a pretrained model does not appear to determine crosslingual performance (cf. appendix b). as models get bigger, only newly pretrained models are available, but at smaller sizes, models modified for cross-lingual alignment perform very well. related languages are more aligned. closely related languages are more aligned within the models. in keeping with the many factors encoded by the representations, it makes sense that as more elements differ between languages, they also become more distant in the representation space. some realignment strategies proved effective at reducing the gap to distant languages, but the pattern remains strong even then. accordingly, some papers applying cross-lingual transfer to specific tasks leverage groups of related languages (e.g., @xcite . the many-to-many translation model m2m-100 @xcite similarly makes use of this by grouping training languages. use available parallel data. while zero-shot cross-lingual transfer is an interesting research problem and can improve pre-trained models, practitioners aiming to deploy the best models should make use of available parallel data. models in the translate-train setting consistently outperform those in the zero-shot setting (cf. appendix b), and even a small number of training examples may help. to what extent is strong alignment necessary? in terms of view i, 'strong' alignment is often seen as desirable, but we question whether this should be a main goal. as discussed in § 2, there is a risk of trading off language-specific information. as for downstream tasks, most cross-lingual retrieval tasks query only the target language space, ignoring the language component. thus, strong alignment may not be necessary for these tasks. lareqa @xcite does test for strong alignment specifically, but it is the exception rather than the rule. in tasks with a prediction head, strong alignment would ensure that the condition of view ii is met. indeed, @xcite show that strong alignment correlates with better performance on downstream tasks, though they specifically look at classification tasks, not retrieval tasks. however, this does not tell us if strong alignment is necessary for improved transfer performance on these tasks, only that it is correlated. ## multilingual generative models recently, the field has turned much attention to generative large language models (llms). we believe multilingual capabilities of generative models will be more and more important as applications scale. thus, we point out several areas of future research, including how cross-lingual alignment may interact with multi-and cross-lingual generation. ## conclusions we have surveyed the literature around crosslingual alignment, providing a taxonomy of methods. we clarified two main views of the concept, noting how simplistic popular measurements are. we summarised insights from the surveyed methods and related analysis papers. going forward, new challenges present themselves with respect to multilingual generative models: simply maximising cross-lingual alignment can lead to wronglanguage generation. we thus call for methods which effectively trade-off cross-lingual semantic information with language-specific axes, allowing models to generate fluent and relevant content in many languages.