Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment (2024)

Abstract: Multilingual generative models obtain remarkable cross-lingual in-context learning capabilities through pre-training on large-scale corpora. However, they still exhibit a performance bias toward high-resource languages and learn isolated distributions of multilingual sentence representations, which may hinder knowledge transfer across languages. To bridge this gap, we propose a simple yet effective cross-lingual alignment framework exploiting pairs of translation sentences. It aligns the internal sentence representations across different languages via multilingual contrastive learning and aligns outputs by following cross-lingual instructions in the target language. Experimental results show that even with less than 0.1 {\textperthousand} of pre-training tokens, our alignment framework significantly boosts the cross-lingual abilities of generative language models and mitigates the performance gap. Further analyses reveal that it results in a better internal multilingual representation distribution of multilingual models.

## introduction multilingual generative language models achieve impressive universality across many languages by pre-training on large-scale unsupervised multilingual corpora @xcite @xcite @xcite . however, models still show a strong language bias toward high-resource languages @xcite , even the state-of-the-art multilingual generative models like gpt-4, exhibiting a 27.5% relative performance gap between english and telugu in mmlu @xcite . this challenge partly arises from the significant linguistic resource imbalance among languages, which is hard to address solely through corpus scaling or balancing. given such a model with language bias and the huge cost of re-training, how can we improve its cross-lingual capabilities and alleviate the language bias using limited data? previous work focused on scaling multilingual instructions @xcite , ignoring internal alignment and knowledge transfer between languages in the multilingual generative model. through visualizing the sentence representations in the multilingual generative model by mean pooling, we find that there is a distinct gap between the sentence representation distributions for different languages like figure 1(a) (the multilingual ones are shown in appendix b.3). this is similar to learning representations for each language separately in the model, which is more challenging for multilingual models to transfer the knowledge learned from other languages. thus, it is important to investigate whether the cross-lingual ability of models will be promoted by learning a better-aligned representation distribution. to address the above issues, we propose a crosslingual alignment framework named align after pre-training (afp), which aims to exploit translation pairs to narrow the gap between languages in the multilingual generation model. to be specific, our method can be divided into the following two modules: 1) multilingual contrastive learning (mcl) on internal representations: we treat a translation sentence pair as the positive example for contrastive learning, and pull the sentence representations in two languages to be closer within the multilingual generated model. this module intends to reduce the differences between languages from the internal representations of the model. 2) cross-lingual instruction following (cif) on the outputs: models must learn to answer in the target language given a prompt from the source language. it aims at enhancing semantic coherence and knowledge transfer across languages in the model. after extensive experiments, it can be found that afp greatly improves the performance of multilingual generative models in cross-lingual natural language inference, multilingual reasoning, and other tasks using less than 1m parallel samples. the performance gap between languages is narrowed, e.g., the relative performance gap of xglm 564m reduces 6.53% on xnli between english and chinese (figure 1(c) ). our method also advances the performance on unseen languages for models, e.g., the chinese performance of llama, which is pretrained on the corpus mainly in english @xcite . further analyses reveal that the representation gap has been mitigated as illustrated in figure 1 (b) after training with afp. in addition, experimental results show that the cross-lingual instruction following task is better than the multilingual instruction tuning task in promoting crosslingual ability with the same parallel corpus. to sum up, our main contributions are as follows: ## method as shown in figure 2 , our framework afp contains the following two modules: 1) multilingual contrastive learning (section 3.1), which aims to align the internal representations of models across different languages. 2) cross-lingual instruction following (section 3.2), which requires models to align the outputs between different languages. ## conclusion and future work in this paper, we proposed a simple yet effective multilingual alignment framework, including internal multilingual representations alignment and cross-lingual outputs alignment methods. experimental results show that this framework improves both the internal representations and cross-lingual capabilities of generative models across various scales. beyond aligning different languages, our framework can be extended to align the internal representations and outputs across different modalities in the multi-modal generative models by replacing parallel samples. however, it is noted that the current framework relies on labeled training data for alignment. future works can focus on the unsupervised multilingual alignment method for language models.