C ross I n: An Efficient Instruction Tuning Approach for Cross-Lingual Knowledge Alignment (2025)

Abstract: Multilingual proficiency presents a significant challenge for large language models (LLMs). English-centric models are usually suboptimal in other languages, particularly those that are linguistically distant from English. This performance discrepancy mainly stems from the imbalanced distribution of training data across languages during pre-training and instruction tuning stages. To address this problem, we propose a novel approach called CrossIn, which utilizes a mixed composition of cross-lingual instruction tuning data. Our method leverages the compressed representation shared by various languages to efficiently enhance the model’s task-solving capabilities and multilingual proficiency within a single process. In addition, we introduce a multi-task and multi-faceted benchmark to evaluate the effectiveness of CrossIn. Experimental results demonstrate that our method substantially improves performance across tasks and languages, and we provide extensive insights into the impact of cross-lingual data volume and the integration of translation data on enhancing multilingual consistency and accuracy.

## introduction the advancement of large language models (llms) like chatgpt @xcite and gemma @xcite has been a game-changer in the field of natural language processing (nlp), revolutionizing tasks such as language generation and commonsense reasoning @xcite . nevertheless, most state-of-the-art llms are english-centric, and their performance on non-english languages is usually suboptimal, especially on languages that are dissimilar to english @xcite @xcite . this challenge mainly stems from the imbalanced distribution of multilingual data at both the pre-training and instruction tuning stages. the exposure bias toward major languages results in an imbalanced capability, where models excel in languages with plentiful data while under-performing in those with limited resources @xcite . bridging the language gap is a fundamental step to unlock the full potential of these general-purpose models and ensure that the benefits are accessible to people across the linguistic spectrum @xcite . efforts to improve the multilingual capabilities of english-centric llms have included continue pre-training using extensive language-specific datasets. yet, mastering languages through additional pre-training could require vast amounts of data and significant computational resources @xcite . on the other hand, despite the limited proportion of non-english data at the pre-training stage, their absolute volume builds a solid knowledge base of various languages. in each iteration, llms are exposed to samples in several languages simultaneously, and the compressed representation encourages models to share linguistic features and generalize across different languages @xcite . however, this ability is not fully retained through the use of datasets that only include english in follow-up tuning steps. in this work, we propose an efficient approach based on a mixed composition of cross-lingual instruction tuning data to exploit llms' underlying multilingual capability, which particularly improves the cross-lingual knowledge alignment @xcite . instruction tuning is to boost the task solving capability of pre-trained language backbones @xcite @xcite . the task and prompt diversity are crucial in both data preparation and the training process, and a small highquality set is sufficient to achieve state-of-the-art zero-shot performance @xcite . however, the language diversity of instruction tuning is often overlooked in english-centric llms. we thus aim to enrich instruction tuning from the language perspective. since all languages share the compressed representation space, cross-lingual instruction tuning can efficiently boost the model's task-solving and multilingual capabilities within a single process. unlike previous work that involved a multi-task setting by adding machine translation and mixing monolingual samples of each language @xcite , we integrate two languages at the sample level and combine various languages at the corpus level. moreover, we compare various mixing strategies to identify the impact of different data formulations. to extensively evaluate the cross-lingual knowledge alignment @xcite , we establish a benchmark of three tasks (i.e., reading comprehension, commonsense questionanswering, and logic reasoning). consistency is measured by analyzing an llm's responses to the same question in different languages, and our benchmark encompasses multiple ability aspects and difficulty levels. moreover, since exact match and f1 score cannot precisely evaluate system outputs in the generative setting, we unify all three tasks in a multiple-choice format for quantitative and reproducible evaluation. the experimental results demonstrate that our mixed cross-lingual tuning can significantly improve performance in all aspects (up to 40% relative gain), followed by a detailed analysis of the influence of data quantity on language consistency and knowledge accuracy. the main contributions of our research are: we introduce crossin, a cross-lingual instruction tuning approach aimed at aligning knowledge across languages to stimulate the model's full multilingual capability after pretraining. it offers a more efficient way of improving the model's performance in various linguistic contexts. ## cross-lingual consistency benchmark since traditional multilingual evaluations often fail to cater specifically to llms or overlook the assessment of cross-lingual consistency in multilingual contexts, in this section, we present a targeted multilingual evaluation benchmark for cross-lingual knowledge alignment. ## crossin method to address the imbalance across languages in english-centric llms pre-training and fine tuning, we explore strategies to enhance multilingual proficiency through cross-lingual instruction tuning. typically, instruction tuning relies on monolingual training samples (e.g., english) @xcite , which limits the potential of massive multilingual exposure at the pre-training stage. therefore, we aim to enrich instruction tuning from the language perspective. since all languages share the compressed representation space, cross-lingual instruction tuning can efficiently boost the model's task-solving and multilingual capabilities within a single process. specifically, we propose crossin approach to mix language compositions at the sample level to enforce the information flow across languages. we hypothesize that our method can further boost multilingual performance by encouraging language-level generalization. the training data can be divided into three main aspects: base, crossin, trans. ## conclusion in this paper, we presented a study on improving cross-lingual knowledge alignment of multilingual large language models, and contributed to both evaluation benchmarks and methodologies. we built a machine comprehension dataset that is a robust resource for extensive multilingual evaluation, emphasizing cross-lingual consistency in compensation with previous datasets. our cross-lingual instruction tuning method crossin brought significant improvements in knowledge accuracy and consistency across languages, highlighting the potential of efficient tuning to create more robust multilingual large language models.