P re A lign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignment (2024)

Abstract: Large language models demonstrate reasonable multilingual abilities, despite predominantly English-centric pretraining. However, the spontaneous multilingual alignment in these models is shown to be weak, leading to unsatisfactory cross-lingual transfer and knowledge sharing. Previous works attempt to address this issue by explicitly injecting multilingual alignment information during or after pretraining. Thus for the early stage in pretraining, the alignment is weak for sharing information or knowledge across languages. In this paper, we propose PreAlign, a framework that establishes multilingual alignment prior to language model pretraining. PreAlign injects multilingual alignment by initializing the model to generate similar representations of aligned words and preserves this alignment using a code-switching strategy during pretraining. Extensive experiments in a synthetic English to English-Clone setting demonstrate that PreAlign significantly outperforms standard multilingual joint training in language modeling, zero-shot cross-lingual transfer, and cross-lingual knowledge application. Further experiments in real-world scenarios further validate PreAlign’s effectiveness across various model sizes.

## introduction large language models @xcite have drastically changed the research paradigm of multilingual language processing. despite being trained on mainly english texts, they still exhibit reasonable ability for other languages @xcite , and have established multilingual alignment to some extent @xcite @xcite . however, researchers @xcite @xcite figure 1 : the illustration of prealign. words in blue, red and green represent translations of piano, guitar and violin, respectively. @xcite have found the spontaneous alignment between languages in these model is still relatively weak, leading to weak cross-lingual factual knowledge retrieval @xcite and inconsistency behaviors given the same input @xcite . a handful of works @xcite @xcite @xcite @xcite try to mitigate the problem by explicitly injecting alignment information using existing supervision data. they either construct cross-lingual prediction tasks @xcite or train models to produce similar representations of aligned words or sentences @xcite @xcite . however, the improvements are somewhat mixed and the establishment of multilingual alignment requires a long training process either during or after pretraining @xcite , which prevents the model from effectively performing cross-lingual transfer at earlier stage in pretraining. in this paper, we introduce prealign, a framework designed to enhance the alignment of pretrained language models. prealign differs from prior methods by integrating the multilingual alignment information before extensive language pretraining and maintaining it throughout the pretraining process. this proactive alignment effectively enhances the learning of cross-lingual knowledge in the pretraining corpus, therefore advancing crosslingual transfer. therefore, model's proficiency in target languages at early training stage is enhanced, leading to improvement of model's ability to acquire knowledge at that stage. more specifically, before large-scale language pretraining, prealign first collects multilingual translation pairs between english and languages to be transferred, and inject this information into the model by pre-pretraining it to produce similar representations of aligned pairs. in order to maintain the established multilingual alignment across the pretraining phase, we propose an inputonly codeswitching strategy, which only substitutes words in the input text to its aligned words, and optimizes model using language modeling objective. the illustration of prealign is shown in figure 1 . we firstly conduct experiments on an english to english-clone setting @xcite @xcite , where english-clone is a synthetic language that shares identical grammar and vocabulary distribution with english, but has no vocab overlap. this allows us to study cross-lingual transfer on a more controlled environment. experiments demonstrate that prealign improves language ability on english-clone by strengthening the cross-lingual transfer of knowledge and abilities from english. further analysis shows that the early established multilingual alignment can be kept throughout large-scale language pretraining and generalize to other unaligned words. experiments on real-world settings (including chinese, german, arabic and russian) validate the effectiveness of prealign across different languages and model scales. our paper differs from theirs in that we focus on improving models' cross-lingual ability and successfully unlocks the ability of cross-lingual knowledge transferring. ## the prealign method in this section, we present prealign, a simple and effective framework that advances the establishment of multilingual alignment before language pretraining. ## evaluation of cross-lingual transfer to investigate the cross-lingual transfer effects that is close to situations in current llms, we design the evaluation in an english-dominated setting, where most of the pretraining data is english. for the examined target language, the amount of pretraining data is much less than english. intuitively, the language ability of the target language will be much weaker than english. however, it is still interesting to know to what extent the language abilities and knowledge could transfer from english to the target language. ## experiments on synthetic setting we start our evaluation on the english to en-clone setting, which allows us to better control the relationship between the source and target language. ## experiments on real-world settings we validate the effectiveness of prealign under real-world settings. performances of lm, zs-clt and clka is shown in prealign are also effective under real-world scenarios. it can be seen from enlarging models is beneficial for clka. we can also see that although joint training gets nearrandom performance at the small scale, the performance grows with the scale of model parameters. this indicates that the ability of spontaneous multilingual alignment only appears on larger models, which is consistent with finding in @xcite . ## conclusion we present the prealign framework in this paper. it advances the establishment of multilingual alignment prior to language pretraining, and maintain it throughout pretraining using an input-only codeswitching strategy. through extensive experiments and analysis, both on synthetic and realworld settings, we demonstrate the effectiveness of prealign for facilitating cross-lingual ability and knowledge transfer.